Ytukyg

When I go to save a JSON-formatted file from inside the Javascript app I'm building (running it on Node v6.11.0 with Express on my local machine, port 3000), I sometimes get the error:

SyntaxError: Unexpected token l in JSON at position 65032, where the "token" is whatever character I assume is the last character before the JSON is truncated, and where the "position" always seems to be "650xx," sometimes as low as 65029, sometimes as high as 65036.

When I delete some content in the file so that it's presumably coming in at less than this length, it saves fine, so that leads me to believe that I'm hitting some sort of default length limit, but from where, I cannot tell. The searching I've done on this question seems to indicate that limits to JSON length are in the MB range at the very least, and the largest file I've saved is 104kB. Meanwhile the limits I'm finding in Node.js, if any are also in the MB range, at least.

The JSON that I'm saving is being generated by the following:

var saveCognitionFile = function( filename, content, url ){



    // Attach the camera state so that we can reinstate it at file load.

    attachCameraStateToCognition();



    var httpRequest = new XMLHttpRequest();



    var body = {};

    var jBody;



    if ( url ){

        body.fullpath = url + '/' + filename;       // filename includes ext        

    }           

    else if ( !url || url === "" ){

        body.fullpath = filename; 

    }



    body.data = content;



    jBody = JSON.stringify( body, circRefFilter );



    // Send the request

    httpRequest.open( "POST", '/saveCognition', true );

    httpRequest.setRequestHeader( "Content-Type", "application/json;charset=UTF-8" );

    httpRequest.send( jBody );  



    debug.master && debug.saveCognition && console.log( httpRequest );



    updateUserFilesList();

}; 

where circRefFilter is an array of strings that get used to filter what key/value pairs get included in the JSON that gets sent.

On the server side then, I do this:

router.post('/saveCognition', function( req, res, next ) { console.log( 'Accessing the Save Cognition route...' );

                        next(); 

                    },

                    function( req, res, next ) { 

                        req.on( 'error', function(){

                            console.log( 'error!' );

                        });

                        req.on( 'data', function( data ) {

                            console.log( 'data at router.post: ', data );



                            var jParsed = JSON.parse( data );

                            console.log( 'data parsed: ', jParsed );



                            jsonMethods.saveCognitionJson( jParsed.fullpath, jParsed.data );

                        });

                        req.on( 'end', function(){

                            console.log( req );

                        });

                        next();

                    },

                    function( req, res ) {

                        res.send('save the current cognition file!'); 

                    } 

                    ); 

Considering that I might be hitting a JSON string length limit of some kind, I do understand that it might make sense to send the file over in chunks, but I'm not sure if that's the solution, or how to go about doing that.

Help appreciated.

Thanks!

UPDATE IN RESPONSE TO Patrick Evans's first offer below.

Probably getting there but... I'm now getting the error

TypeError: Cannot read property 'length' of undefined

    at Function.Buffer.concat (buffer.js:304:24)

    at IncomingMessage.<anonymous> (C:UsersMarkdocumentspermiebotresearchexperimentsthree-jslucidnodesroutes.js:52:24)

    at emitNone (events.js:86:13)

    at IncomingMessage.emit (events.js:185:7)

    at endReadableNT (_stream_readable.js:974:12)

    at _combinedTickCallback (internal/process/next_tick.js:80:11)

    at process._tickCallback (internal/process/next_tick.js:104:9)

Here's the updated server-side code:

router.post('/saveCognition', function( req, res, next ) { console.log( 'Accessing the Save Cognition route...' );

                        next(); 

                    },

                    function( req, res, next ) { 



                        let reqData = ;

                        let buffer;



                        req.on( 'error', function(){

                            console.log( 'error!' );

                        });

                        req.on( 'data', function( data, chunk ) {



                            reqData.push( chunk );



                            console.log( 'data at router.post: ', data );



                        });

                        req.on( 'end', function(){



                            buffer = Buffer.concat( reqData ).toString();



                            var jParsed = JSON.parse( /* data */ buffer );

                            console.log( 'data parsed: ', jParsed );



                            jsonMethods.saveCognitionJson( jParsed.fullpath, jParsed.data );                            



                            console.log( req );

                        });

                        next();

                    },

                    function( req, res ) {

                        res.send('save the current cognition file!'); 

                    } 

                    );

Apparently I'm not concatenating correctly. removing .toString() doesn't help. Thanks.

Well the UPDATE with the second error turned out to be a simple fix. See my answer below. Thanks to @PatrickEvans for help on this in the comments.

With help from @PatrickEvans in the comments on my question and help from an outside tutorial here https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/ I was able to figure this out.

As Patrick said:

.on('data'... gets called multiple times (ie your data comes in
chunks), keep concatenating data then do the parse in .on('end'

It took me a minute to figure out how to do the concatenation, but here's the relevant working server-side code.

router.post('/saveCognition', function( req, res, next ) { console.log( 'Accessing the Save Cognition route...' );

                        next(); 

                    },

                    function( req, res, next ) { 



                        let reqData = ;

                        let buffer;



                        req.on( 'error', function(){

                            console.log( 'error!' );

                        });

                        req.on( 'data', function( data ) {



                            reqData.push( data );

                            console.log( 'data at router.post: ', data );



                        });

                        req.on( 'end', function(){



                            buffer = Buffer.concat( reqData ).toString();



                            var jParsed = JSON.parse( buffer );

                            console.log( 'data parsed: ', jParsed );



                            jsonMethods.saveCognitionJson( jParsed.fullpath, jParsed.data );                            



                            console.log( req );

                        });

                        next();

                    },

                    function( req, res ) {

                        res.send('save the current cognition file!'); 

                    } 

                    );