Serial LINQ Queries

I am trying to create a LINQ query to query a large SQL table (7M + records) Documents

.

Each document has many DocumentFields

:

Simplified UML

My goal is to apply sequential filters (0 to ~ 10 filters) on the field value

DocumentField

:

Here's an example of the filters I want to apply:

[
  {fieldId: 32, value: "CET20533"},
  {fieldId: 16, value: "882341"},
  {fieldId: 12, value: "101746"}
]

      

I want to get every document in my database that matches all filters. In the previous example, I want all documents that have CET20533 for field ID "32", value 882341 for field ID 16, and so on.

I had my first approach:

List<MyFilter> filters = ... // Json deserialization
db.Documents.Where(document =>
  filters.All(filter =>
     document.DocumentFields.Any(documentField =>
        documentField.Id == filter.Id
        && documentField.Value == filter.Value)));

      

This approach does not work: my filters

List is not a primitive type and therefore cannot be used in a LINQ query.

I had a second approach which didn't throw me errors, but only applied 1 filter:

var result = db.Documents.Select(d => d);
foreach (var filter in filters) {
  var id = filter.Id;
  var value = filter.Value;

  result = result.Where(document => document.DocumentFields.Any(documentField =>
    documentField.Id == id
    && documentField.Value == value));
}

      

The problem with this approach, in my opinion, is the concurrency issue. I have applied a simple pause Thread.Sleep(2000)

on each iteration of the foreach test for testing and it seems to work.

Questions:

  • How do I remove the pause and still not have concurrency issues?
  • Is there a better way to create my request?

EDIT:

For clarity, here's an actual example of a document that matches the example of the previous filters:

Sample document

+3


source to share


3 answers


Strongly convinced that your data model is too general. This can hurt you in terms of the clarity and performance of the program.

But release it for this answer, which I took as a challenge in constructing expressions. The goal is to get a good query that distinguishes filters on the data server side.

Here is the data model I used that I think matches yours:

public sealed class Document
{
    public int Id { get; set; }
    // ...
    public ICollection<DocumentField> Fields { get; set; }
}

public sealed class DocumentField
{
    public int Id { get; set; }
    public int DocumentId { get; set; }
    public string StringValue { get; set; }
    public float? FloatValue { get; set; }
    // more typed vales here
}

      

First, I implement convenience functions to create predicates for individual fields of specific field types:

public static class DocumentExtensions
{
    private static readonly PropertyInfo _piFieldId = (PropertyInfo)((MemberExpression)((Expression<Func<DocumentField, int>>)(f => f.Id)).Body).Member;

    private static Expression<Func<DocumentField, bool>> FieldPredicate<T>(int fieldId, T value, Expression<Func<DocumentField, T>> fieldAccessor)
    {
        var pField = fieldAccessor.Parameters[0];
        var xEqualId = Expression.Equal(Expression.Property(pField, _piFieldId), Expression.Constant(fieldId));
        var xEqualValue = Expression.Equal(fieldAccessor.Body, Expression.Constant(value, typeof(T)));
        return Expression.Lambda<Func<DocumentField, bool>>(Expression.AndAlso(xEqualId, xEqualValue), pField);
    }

    /// <summary>
    /// f => f.<see cref="DocumentField.Id"/> == <paramref name="fieldId"/> && f.<see cref="DocumentField.StringValue"/> == <paramref name="value"/>.
    /// </summary>
    public static Expression<Func<DocumentField, bool>> FieldPredicate(int fieldId, string value) => FieldPredicate(fieldId, value, f => f.StringValue);

    /// <summary>
    /// f => f.<see cref="DocumentField.Id"/> == <paramref name="fieldId"/> && f.<see cref="DocumentField.FloatValue"/> == <paramref name="value"/>.
    /// </summary>
    public static Expression<Func<DocumentField, bool>> FieldPredicate(int fieldId, float? value) => FieldPredicate(fieldId, value, f => f.FloatValue);

    // more overloads here
}

      

Using:



var fieldPredicates = new[] {
        DocumentExtensions.FieldPredicate(32, "CET20533"), // f => f.Id == 32 && f.StringValue == "CET20533"
        DocumentExtensions.FieldPredicate(16, "882341"),
        DocumentExtensions.FieldPredicate(12, 101746F) // f => f.Id == 12 && f.FloatValue == 101746F
};

      

Second, I am implementing an extension method HavingAllFields

(also in DocumentExtensions

) that creates IQueryable<Document>

where all of the field's predicates are satisfied by at least one field:

    private static readonly MethodInfo _miAnyWhere = ((MethodCallExpression)((Expression<Func<IEnumerable<DocumentField>, bool>>)(fields => fields.Any(f => false))).Body).Method;
    private static readonly Expression<Func<Document, IEnumerable<DocumentField>>> _fieldsAccessor = doc => doc.Fields;

    /// <summary>
    /// <paramref name="documents"/>.Where(doc => doc.Fields.Any(<paramref name="fieldPredicates"/>[0]) && ... )
    /// </summary>
    public static IQueryable<Document> HavingAllFields(this IQueryable<Document> documents, IEnumerable<Expression<Func<DocumentField, bool>>> fieldPredicates)
    {
        using (var e = fieldPredicates.GetEnumerator())
        {
            if (!e.MoveNext()) return documents;

            Expression predicateBody = Expression.Call(_miAnyWhere, _fieldsAccessor.Body, e.Current);
            while (e.MoveNext())
                predicateBody = Expression.AndAlso(predicateBody, Expression.Call(_miAnyWhere, _fieldsAccessor.Body, e.Current));
            var predicate = Expression.Lambda<Func<Document, bool>>(predicateBody, _fieldsAccessor.Parameters);
            return documents.Where(predicate);
        }
    }

      

Test:

var documents = (new[]
{
    new Document
    {
        Id = 1,
        Fields = new[]
        {
            new DocumentField { Id = 32, StringValue = "CET20533" },
            new DocumentField { Id = 16, StringValue = "882341" },
            new DocumentField { Id = 12, FloatValue = 101746F },
        }
    },
    new Document
    {
        Id = 2,
        Fields = new[]
        {
            new DocumentField { Id = 32, StringValue = "Bla" },
            new DocumentField { Id = 16, StringValue = "882341" },
            new DocumentField { Id = 12, FloatValue = 101746F },
        }
    }
}).AsQueryable();
var matches = documents.HavingAllFields(fieldPredicates).ToList();

      

Corresponds to document 1, but not 2.

+1


source


You should create expressions based on your filters and add them where they are (or not, if you can control it)

db.Documents.Where(ex1).Where(ex2)...

      



see eg and MSDN

Or a simple case: start with DocumentFields and find related documents. Activity Contains activities for simple types. which will also be easier in the case of constructing an expression

+1


source


I usually do something like this: put in your list all the needed ids for your filter, then use contains

.

List<int> myDesiredIds = new List<int> { 1, 2, 3, 4, 5 };
db.documents.Where(x=>myDesiredIds.Contains(x.DocumentId));

      

0


source







All Articles