MongoError on file upload using mongoose, gridfs-stream and multer

I am running express 4 using multer , gridfs-stream and mongoose with mongodb and I am trying to download a file and pipe it to gridfs.

The express route that does this is defined as:

app.post('/uploadfile', function (req, res) {
    console.dir(req.files);

    // The mongodb instance created when the mongoose.connection is opened
    var db = mongoose.connection.db;

    // The native mongo driver which is used by mongoose
    var mongoDriver = mongoose.mongo;

    // Create a gridfs-stream
    var gfs = new Gridfs(db, mongoDriver);

    var file = req.files.myFile;

    var fileId = new ObjectId();

    console.log("Creating WriteStream");
    var writeStream = gfs.createWriteStream({
        _id: fileId,
        filename: file.originalname,
        mode: 'w',
        content_type: file.mimetype,
        metadata: {
            id: '123',
            number: '2',
            name: "Kenny Erasmuson"
        }
     });
     console.log("Created WriteStream");
     req.pipe(writeStream);
     console.log("Finished!");
});

      

Once the express app is running, the file is selected and loaded (via the HTML / multipart / form-data form) and the output from the node server is:

$ node server.js 
Listening on port 8001
{ myFile: 
    { fieldname: 'myFile',
    originalname: 'kenny-credit-rating.pdf',
    name: '1082e5071ede1002c4ae5be6123226d8.pdf',
    encoding: '7bit',
    mimetype: 'application/pdf',
    path: 'uploads/1082e5071ede1002c4ae5be6123226d8.pdf',
    extension: 'pdf',
    size: 110782,
    truncated: false,
    buffer: null } 
}
Creating WriteStream
Created WriteStream
Finished!
/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/co nnection/base.js:246
    throw message;      
          ^
MongoError: The dollar ($) prefixed field '$conditionalHandlers' in '_id.$conditionalHandlers' is not valid for storage.
at Object.toError (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/utils.js:114:11)
at /Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/collection/core.js:569:27
at /Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1157:7
at /Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/db.js:1890:9
at Server.Base._callHandler (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/base.js:448:41)
at /Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:481:18
at MongoReply.parseBody (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
at null.<anonymous> (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:439:20)
at emit (events.js:95:17)
at null.<anonymous> (/Users/kenny/Dropbox/Dev/fileUploader/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:201:13)

      

Does anyone have any ideas what is causing the error and how to fix it?

+3


source to share


2 answers


In my code:

var fileId = new ObjectId();

console.log("Creating WriteStream");
var writeStream = gfs.createWriteStream({
    _id: fileId,
    filename: file.originalname,
    mode: 'w',
    content_type: file.mimetype,
    metadata: {
        id: '123',
        number: '2',
        name: "Kenny Erasmuson"
    }
 });

      

I am assigning an ObjectId, not a string representation of the ObjectId, to the _id property of the object passed to gfs.createWriteStream. It turns out this is throwing MongoError in my code.



The fix I found here is to change the line of code causing the problem:

_id: fileId.str,

      

Having done this, I ended up with the question that the file was not put into the ms.ods fs.chunks collection, although the metadata ends up in the fs.files collection of mongodb. AddieD's answer here starts to address this :)

+2


source


The problem is that you are not only passing the file-request to the gridfs-stream. Using Multer as middleware catching any multipart/form-data

post request . As the stream arrives, Multer (built on Busboy ) observes the events on('field')

and on('file')

analyzes accordingly. What you are linking to Multer is not just a file.

This bit of code works great because Multer has really figured you out req.files

and req.body

at this point:

   // Create a gridfs-stream
   var gfs = new Gridfs(db, mongoDriver);

   var file = req.files.myFile;

   var fileId = new ObjectId();

   console.log("Creating WriteStream");
   var writeStream = gfs.createWriteStream({
        _id: fileId,
        filename: file.originalname,
        mode: 'w',
        content_type: file.mimetype,
        metadata: {
            id: '123',
            number: '2',
            name: "Kenny Erasmuson"
        }
     });
     console.log("Created WriteStream");

      

But where your code runs into problems, here's for the reasons mentioned above:

 req.pipe(writeStream);

      

I still need to find a way to stream a multipart / form-data post request with more than a simple file upload right to GridFS. If your post request contains nothing but a file (which means there are no other html input fields in the form), you might want to want to remove Multer from the middleware at least for this route.



In my use case, I need to be able to receive an html message with text inputs along with the file upload (I store metadata about the file on upload). This is how I accomplished what I needed with Multer:

var uploadImg = function(req,res) {
      var writestream = gfs.createWriteStream({
        filename: req.files.file.name,
        mode:'w',
        content_type:req.files.file.mimetype,
        metadata:req.body,
      });
    fs.createReadStream(req.files.file.path).pipe(writestream);

    writestream.on('close', function (file) {
        res.send("Success!");
        fs.unlink(req.files.file.path, function (err) {
          if (err) console.error("Error: " + err);
          console.log('successfully deleted : '+ req.files.file.path );
        });
    });

};

      

By default Multer will store your files on disk. One quick solution is to simply create a read stream and pipe it back to GridFS. When finished, delete the tmp file.

As a side note, there seems to be some thought that it is better to use Multer as middleware only on routes that need it. You can read more about this thought in the last section of this .

Update

I think I'm close to finding a way to stream directly into GridFS with metadata using Skipper . I'm in the process of browsing if I can't get some updates going to skipper-gridfs that will accept html form text inputs and set them as metadata, etc. Setting up the shadow skipper system seems pretty minor. I'll update when this goes off. If you look at the skipper, make sure you understand what order of your html form inputs (if you have one) matters.

+5


source







All Articles