How can I speed up scanning a large string with a large regex?

Lets assume that order_ids

is an array of order IDs that I assume contains numbers and / or digits and case is not an issue. If email

is a string containing one email,

email.scan(/\w+/) & order_ids

      

        
        
        
      

    

will provide you with all the order IDs contained in this email. However, you can speed this up by first converting order_ids

to a set:

require 'set'
order_id_set = order_ids.to_set

email.scan(/\w+/).select { |w| order_id_set.include?(w) }

      

        
        
        
      

    

There is no advantage to stacking letters together anyway.

I suggested that you split the letter into words using:

email.scan(/\w+/)

      

        
        
        
      

    

but you may need something more complicated. Let's say the email was as follows:

email = "I am really annoyed that my order AB123 was shipped late, " +
  "that order CD456was poorly packed and I was overcharged for order EF789."

      

        
        
        
      

    

Let's look at two ways to break it down into words:

esplit = email.split
  #=> ["I", "am", "really", "annoyed", "that", "my", "order", "AB123",
  #    "was", "shipped", "late,", "that", "order", "CD456was", "poorly",
  #    "packed", "and", "I", "was", "overcharged", "for", "order", "EF789."]

escan = email.scan(/\w+/)
  #=> ["I", "am", "really", "annoyed", "that", "my", "order", "AB123",
  #    "was", "shipped", "late", "that", "order", "CD456was", "poorly",
  #    "packed", "and", "I", "was", "overcharged", "for", "order", "EF789"]

      

        
        
        
      

    

You can see that the problem is welded (for example "CD456was"

) with both of these approaches. Now let's create a set of order IDs:

require 'set'
order_id_set = %w{ AB123 CD456 EF789 GH012 }.to_set
  #=> #<Set: {"AB123", "CD456", "EF789", "GH012"}>

      

        
        
        
      

    

and then do an email search for order IDs:

esplit.select { |w| order_id_set.include?(w) }
  #=> ["AB123"]
escan.select  { |w| order_id_set.include?(w) }
  #=> ["AB123", "EF789"]

      

        
        
        
      

    

As you can see, split

(the same as split(' ')

and split(/\s+/)

), which splits the string on spaces, caught the first order ID, but missed the other two, CD456

because the writer was unable to establish a gap between the id and the word "was"

, and EF789

, because he was looking for "EF789."

.

scan(/\w+/)

That breaks the "word" ( \w

searches for a-z

, a-z

, 0-9

and _

), better by identifying "EF789"

how to order ID, because the period is a symbol of "non-word". He also missed "CD456"

, however, since "w"

, "a"

and "s"

are symbols of words, so they included them. (This is similar to your example "1234"

.

This means you need to create a better regexp. If, for example, all the order IDs were similar to those in my example - two capital letters followed by three numbers, you can write:

email.scan(/(?<![A-Z])[A-Z]{2}\d{3}(?!\d)/)
     .select { |w| order_id_set.include?(w) }
  #=> ["AB123", "CD456", "EF789"]

      

        
        
        
      

    

This matches two uppercase letters, preceded by an uppercase letter, followed by three numbers, followed by any character other than another digit. This was just an illustration of what is possible. You might want to post another question regarding solely the use of regular expressions to determine possible order numbers in emails. To do this, of course, you'd better describe what the serial numbers look like.

In any case, your regex will not contain the #{order_id}

one you mentioned. The only way to do this would be to check each email with a separate regex for each order ID, which would be extremely inefficient. Instead, you want to use a regular expression to determine the possible order numbers and then test them against a set of order numbers. When choosing a regular expression, you may have a trade-off between the number of missing order IDs and the number of custom ID lines that you spend time validating a set of order IDs.