how to get multiple matches with difflib.SequenceMatcher?

difflib, python, regex

Solution

As Jerry pointed out, and k-nut correctly answers, you are using the wrong algorithm for your problem. k-nut's answer honestly isn't all that bad, but it's not exactly the most efficient way to solve this class of problems. I am a bioinformatician and given your question and the example case, it seems very much like you are trying to solve "our" classical DNA sequence alignment/search problem (see the scientific literature by scientific super-stars like Altschul or "Gene" Myers on the issue if you are interested in the nitty-gritty details and want to read one of the most cited papers of all times).

Finding short segments in a database of long segments efficiently is exactly what Altschul's now famous BLAST algorithm solves heuristically and/or can be done using Smith-Waterman for exact lookups. The most efficient way to do this in Python is probably by using BioPython, and in particular, you might want to look at the section describing how to set up a local NCBI BLAST+ instance. If you are not "married" to Python, today there are even faster implementations of BLAST, like FSA-BLAST.

On the other hand, if you need exact matches (as opposed to the heuristics made by BLAST), which might be the case if you don't mind long query times and have a small reference sequence (`B` in your examples), you could go with the official Smith-Waterman (SW) alignment. If not, and you still need exact matches, first filter matches with BLAST and then reduce your set with SW alignments of the candidates.

You could implement SW in pure Python, even just use any existing pure-Python implementation, but I would only recommended that path for purely educational purposes (check out swalign on GitHub, for example). If you nonetheless want a reasonably strong Python-based implementation check out scikit-bio for SW alignments, although scikit-bio is still in alpha-status. But first read up on the SW WikiPedia page already linked above, and depending on the hardware you have, you might instead use a GPU- or at least SIMD-optimized implementation in CUDA or C++. If you want a nice version with a Python wrapper, check out the SSWlib.

Problem

I am using difflib to identify all the matches of a short string in a longer sequence. However it seems that when there are multiple matches, difflib only returns one: ``` > sm = difflib.SequenceMatcher(None, a='ACT', b='ACTGACT') > sm.get_matching_blocks() [Match(a=0, b=0, size=3), Match(a=3, b=7, size=0)] ``` The output I expected was: ``` [Match(a=0, b=0, size=3), Match(a=0, b=4, size=3), Match(a=3, b=7, size=0)] ``` In fact the string ACTGACT contains two matches of ACT, at positions 0 and 4, both of size 3 (plus another match of size 0 at the end of the strings). How can I get multiple matches? I was expecting difflib to return both positions.

Original source