OnlyOffice Documents Not Saved In Nextcloud: How to Recover

While recently working on my grad school application essay in OnlyOffice, I suffered a catastrophic file loss. Here’s what happened (and more importantly, how I recovered).

Update: 3/10/21

There is another option that may help those of you who don’t have the option of being able to fall back to a versioned backup. You could be lucky and all you need to do is flush the Document Server so that it pushes/commits your edits to file. Try this first:

occ documentserver:flush

  • The above applies to a native/local install, where occ shoul be recognized as a command. If you’re in a Docker container, you may need to use a command like this instead:

/var/www/html/occ documentserver:flush

I had the problem again this weekend and the above worked for me. (See reply to this post for root cause investigation and how you can prevent this problem going forward in the future). If this doesn’t work for you, you may have to proceed to Plan B below.


NOTE: The only way I was able to recover is because I have a robust backup system in place on my server that automatically gives me versioned backups. You will need a versioned backup to restore from.

The Problem

  • I renamed my document in Nextcloud.
  • Tried to open the newly renamed document, OnlyOffice attempted to open the document, and errors out, telling me that I should download a copy and then open the file.

OnlyOffice Nextcloud Error: “An error occurred during the work with the document. Use the ‘Download as…’ option to save the file backup copy to your computer hard drive.”

  • I download a copy of the file and open it in Word and…the document is empty!

Recovery Attempts

  • Thankfully, I run Crashplan and keep versioned backups of my documents.
  • I download yesterday’s backup and attempt to open it in Word and…this file too is empty!

Analysis/Troubleshooting

  • The reason this document is blank, even the backup, is because the file is truly blank!
  • How is this possible? I’ve been editing the document across multiple browser sessions over several days. It was there just an hour ago.
  • Conclusion: OnlyOffice hasn’t been committing the changes to the document itself.

Researching the issue, it seems this is well-known issue. Mind boggling that NextCloud/OnlyOffice would allow such a dangerous issue to stand, but apparently they do:

Heck, you can even see it in the Document Server review:

https://apps.nextcloud.com/apps/documentserver_community

Okay, so if OnlyOffice isn’t committing the changes to document file itself, where does it store them? Well, I recall in setting this up that there was a document server in use. Going off of that, I happened to stumble upon this clue which I will copy here just in case it is ever lost:

Edits are saved in the MySQL/MariaDB Table oc_documentserver_changes

Source: dreistreifenliebe at Onlyoffice documents not saved in NextCloud - #6 by dreistreifenliebe - community document server - Nextcloud community

I was very afraid that OnlyOffice stored everything in the browser’s cache, but thankful that this did not appear to be the case.

Recovering with “The Fix”

Armed with this new piece of information, I did the following:

  1. Reverted the name of the renamed file back to its original name.

  2. In Crashplan, I opened up the MariaDB Docker container’s bind mount on my where my Nextcloud databases are stored.

  3. I changed the date in Crashplan to the day before (my last guaranteed known-good date) and selected all files beginning with oc_documentserver_* and oc_onlyoffice_* for restoration.

  • Note that this was likely overkill and restoring just oc_documentserver_changes would’ve been sufficient, but this was a critical situation and I was under a time crunch, so I didn’t have the luxury of time to play around.
  1. I restarted the Nextcloud container.

  2. I went back to the file in question in Nextcloud and attempted to open it. It still gave me the same error, but this time, when I selected the “Download as…” option, the document actually downloaded and wasn’t empty.

Hope this helps someone out,
TorqueWrench

Root Cause Investigation For Nextcloud Not Committing OnlyOffice Changes to File

So I noticed while working on a document in OnlyOffice last night that yet again Nextcloud’s Document Server had, yet again, not actually committed the changes to the file. The contents and everything were still there when I viewed the document in OnlyOffice, but the document’s modified timestamp had not been updated…a red flag!

It has been suggested in many Nextcloud Git issues that the way to prevent this is to “nicely” close the document with the “Open file location” button, but I can confirm that this does nothing, at least for my issue. Doing so did not result in the timestamp updating.

So this time I attempted to force Nextcloud’s Document Server to commit the changes to file with ./occ documentserver:flush inside the container. This fixed the issue of committing the modifications to the file itself. (It also brought a bunch of “old files” that I hadn’t worked on in a week back to the surface as recently modified in my dashboard, interestingly restoring my “lost” grad school application essays that I had recovered through alternate methods. See post #1 above.)

Document Server Background

So the above only “fixes” the issue after the problem has already happened. It doesn’t prevent the problem going forward and so the underlying cause is still a very real threat to data integrity. Continuing to dig around, I found further complaints of this problem in Nextcloud’s Document Server git issues, but this one for some reason caught my eye:

Especially this post:

robinm8 suggested that cron jobs were at play here and that made perfect sense to me that the Document Server would flush its own document modifications at a set interval with a cron job. So I checked my Nextcloud container’s “Basic Settings” and saw this:

Unfortunately, I don’t know if it was set to “cron” or “AJAX” originally. This is the image order I have saved off, but I may have been switching them back and forth. Regardless, the real issue as you can see is that the Nextcloud cron/AJAX jobs haven’t executed in over two months!

The problem is that basic background jobs aren’t running.

Interestingly, cycling between cron and the AJAX jobs didn’t fix anything. AJAX should theoretically make the background jobs run with each page load (which I think is ideal if you’re actually using OnlyOffice in the browser). I believe this is also why so many of these issue reports attribute the problem to not “properly closing” the document as mentioned above: properly closing the document triggers a new page load which would trigger the AJAX background job to flush the Document Server and commit the changes.

So how did I fix this? Well, for some reason (and I’m still not entirely sure why), I had to manually run a cron job with php /var/www/html/cron.php. After that, switching to AJAX and monitoring the background jobs in “Basic Settings” showed the background jobs executing with each page load. It was also further confirmed by the updating of the document timestamps.

I also know why the cron job isn’t running when I choose that option but that’s specific to my server configuration on Unraid. For other users, who fall into the bucket of “not using AJAX and instead using cron” but are still running into this problem, I suspect their issue is with the configuration of the user they’re using for their Docker container- specifically that it’s not www-data which is the only user that the Nextcloud Docker’s busybox is set up to run a cron job for.

Again, I hope this helps everyone else in the Nextcloud/OnlyOffice community. I know how scary data corruption/loss can be and if this can help just one person, it was worth it.

-TorqueWrench

Update 3/11/21:

I went to bed with Nextcloud working perfectly with OnlyOffice. Woke up this morning and was greeted with this:

Presumably, after some period of inactivity in Nextcloud, the background jobs get “hung up” and have to be restarted with a forced cron job using php /var/www/html/cron.php . That seems like a real issue; I’ll submit an issue to Nextcloud about it.

Edit: Issue opened with Nextcloud/docker: Issue 1442

For now, I’ve settled on a cron job on the Docker host itself to execute a script every five minutes. Stay tuned!

-TorqueWrench

Additional Notes: 3/13/21

While I now have a cron task set up to run q5min to run both the Nextcloud cron.php background job and set it back to the AJAX job, and the AJAX background jobs are being reported as running, I’ve noticed that the document files are not updating until the cron.php job itself does.

So it appears that the Nextcloud Document Server is not flushing with the AJAX background job, only the cron.php job. I suppose I’ll have to open up another issue with that on Nextcloud/documentserver_community…