Make sure we return all of the relevant results, even in cases with
lots of results (larger than ENTR_PER_PAGE in the offset index) and
where you have a synyonym and headword present for the same word.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
They aren't used at all by scdv, and thus aren't tested (meaning that
adaptions to the core lookup algorithms can be complicated because these
methods use them but aren't tested so there's no real way of knowing if
a change has broken the methods or not).
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Previously, we would just return the first entry we found that matched
the requested word. This causes issues with dictionaries that have lots
of entries which can be found using the same search string. In these
cases, the user got a completely arbitrary word returned to them rather
than the full set.
While this may seem strange, this is incredibly commonplace in Japanese
and likely several other languages. In Japanese:
* When written using kanji, the same string of characters could refer
to more than one word which may have a completely different meaning.
Examples include 潜る (くぐる、もぐる) and 辛い (からい、つらい).
* When written in kana, the same string of characters can also refer to
more than one word which is written using completely different kanji,
and has a completely different meaning. Examples include きく
(聞く、効く、菊) and たつ (立つ、建つ、絶つ).
In both cases, these are different words in every sense of the word, and
have separate headwords for each in the dictionary. Thus in order to be
completely useful for such dictionaries, sdcv needs to be able to return
every matching word in the dictionary.
The solution is conceptually simple -- return a set containing the
indices rather than just a single index. Since every list we search is
sorted (to allow binary searching), once we find one match we can just
walk backwards and forwards from the match point to find the entire
block of matching terms and add them to the set in linear time. A
std::set is used so that we don't return duplicate results needlessly.
This solution was in practice a bit more complicated because .otf cache
files require a bit more fiddling, and also the ->lookup methods are
used by some callers to find the next entry if no entry was found. But
on the whole it's not too drastic of a change from the previous setup.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
Due to the lack of deinflection support in StarDict, users might want to
be able to create a list of possible deinflections and search each one
to see if there is a dictionary entry for that deinflection.
Being able to do this in one sdcv invocation is far more preferable to
calling sdcv once for each candidate due to the performance cost of
doing so. The most obvious language that would benefit from this is
Japanese, but I'm sure other folks would prefer this.
In order to make this use-case better supported -- try to look up every
word in the provided list of words before existing with an error if any
one of the words failed to be looked up.
Signed-off-by: Aleksa Sarai <cyphar@cyphar.com>
the previous approachonly works as long as locales are actually sane
(i.e., the test only passes if you *actually* have the ru_RU.KOI8-R
locale built, which the CI doesn't).
This allows to detect different endianity of machines to avoid
loading caches created with different endianity.
Fixes#36
Signed-off-by: Michal Čihař <michal@cihar.com>
