MongoDB update request is too long

Question

MongoDB update request is too long

We have a shard cluster with 4 mongos, 3 configurations, and two shards (each with a 3 member replica set). All instances of mongo are on version 3.2.

We have a timely job done every 5 minutes that picks up (adds if not, and updates if present) approximately 16,000 records in a collection currently with 150 million records. This collection has been kept for the last couple of years and there are about 20,000 new inserts every day.

Until the last day (yesterday) this update process took only 30-35 seconds. But now it is observed that this update time suddenly increases to about 110 seconds.

We have used the following methods to track down the cause, but could not find the root cause -

Configuration servers are synchronized. (db.runCommand ('dbhash') on config servers)
No other operations / tasks have been added / changed in this time window.
The replica or cluster configuration has changed.
The balancer picks up and releases the lock as expected.
The target collection is almost equally distributed on both shards (73 million on one and 77 million on the others).
The average load for all linux boxes is 1.0.

We are stuck and cannot find the cause of this sudden firing in script time. Any help is appreciated.

Insert and update code -

    //dataMap is a hashMap containing all parameters.
    UpdateOptions updateOptions = new UpdateOptions().upsert(true);
    Document updateObject = new Document("$setOnInsert",new Document("cat", dataMap.get("cat"))
                            .append("$set", new Document("timeStamp",dataMap.get("timeStamp"))));
    Document query = new Document("host",dataMap.get("host"));                              
    query.append("timeStamp",dataMap.get(timeStamp));
    // collectionObject below is object of MongoCollection<Document>
    collectionObject.updateOne(query, updateObject, updateOptions);