Computing a score from multiple classifiers

I am trying to determine the similarity between pairs of items taken from a large collection. Elements have multiple attributes and I can calculate a discrete similarity score for each attribute, between 0 and 1. I use different classifiers depending on the attribute: TF-IDF cosine similarity, Naive Bayes classifier, etc.

I got stuck when it came to compiling all this information into a final similarity score for all elements. I can't just take the unweighted mean because 1) that the high score depends on the classifier, and 2) some classifiers are more important than others. In addition, some classifiers should only be counted for their high scores, that is, a high score indicates higher similarity, but lower scores are irrelevant.

So far, I have calculated the final result with guesswork, but an increasing number of classifiers make this a very bad decision. What methods are there for determining the optimal formula that will take my different estimates and return only one? It is important to note that the system does receive human feedback, which some classifiers start with.

Ultimately, I'm only interested in the rating, for each element the most similar. Absolute scores are meaningless in themselves, only ordering is important.