My suggestion for your case is: change to use external data Run the move tool to move files from /internal_data to /data Upload the files (from /data) to S3 with Amazon command line tool Run the storage options tool to update links from /data to S3 Otherwise it won't works since your files are too big to be handled by PHP script. Sorry, Dropbox is currently not supported.
Hi, Are you ever going to be able to provide help when someone has an issue when trying to correct the links to their attachments after using this add-on? I still have the problem where I changed servers and have a new ftp username, etc. and when I try to rebuild those old attachments and/or thumbnails that were uploaded with my old ftp username and etc. it won't do it. It always says it cannot log on using that old username and etc. when trying to rebuild them. Is there a way to run an SQL query or something to where we can somehow update those old stored ftp usernames, ports, and passwords? In my case I have only used the ftp feature in this add-on. .
xfrocks @rsi The add-on comes with a tool in AdminCP > Tools > Rebuild Cache to update storage options. If you ever move your files manually (to/from S3/FTP etc.) you can use the tool to update the url selectively to moved attachments. Let me know if it doesn't work for you?
rsi @xfrocks Hi, I've tried that and it doesn't work. All of my attachments were moved to another folder on my server using the ftp option in the add-on. I've never used it for s3. What happened is when I moved everything to another server (not with the attachment store add-on of course) the new hoisting company gave me a different ftp username and password than what I had before on my old server. So when I try to use those rebuild options in the ACP as you have suggested it get thousands of errors because it says it cannot log in with that old username, etc. that were attached to those attachments using the attachment store add-on when I had that old username, etc. on my old server. It rebuilds everything with the new username, etc. but it just cannot do the old attachments and the hosting company will lock me out as a security risk because of thousands of failed login attempts when it tries to rebuild those old attachments. Apparently the attachment store add-on stores your ftp username and maybe the password and the port number when it moves attachments. I know it at least stores the username. I'm assuming it’s in the hash, but I'm not positive. It there a way you can give me an SQL query or a script I can run to update all of those old usernames and whatever else the attachments have stored like maybe the password, port number, etc? Either that or add a manual update rebuild feature to the add-on itselt that would do the same thing? Thanks… .
xfrocks @rsi There are 2 tools, the one you mentioned and tried is the move tool. There is another tool named "Update Attachment Data Storage Options" which you should use to update the saved information and point them to the new server since you moved the files yourself.
rsi @xfrocks You mean this one below right? I've tried that before and it didn't work. Is there some special setting? I think I tried it with 'Store file in remote server via FTP', but I believe I also tried 'Default' and 'External' as well. And with 'Store file in remote server via FTP' I'd use my current ftp username and password and not the old one right? All of my files are also in the same folder as they have always been so would maybe creating a new folder and moving them to it with that rebuild process do it if that is the problem? Whenever I have tried it I get thousands of errors for each of the old attachments saying it could not log in with the username and etc. attached to those old attachments. Also, do any of the settings below need to be changed as well while doing it? Below is what I basically have. .
Yes, that's the below tool. Basically you need to update your new ftp server on the Options page (2nd screenshot) then use the tool (1st screenshot) and enter your old ftp server info. You don't need to enter all info btw. For example: if you have only used 1 server, entering the old server host is enough because the tool will look for existing attachments that match the information you provided and update them with new configuration (as in Options page).
rsi @xfrocks I'll need to try it again and see what happens, but I'm actually back with my old hosting company and on another server which is the 3rd one now, but I no longer have any traces of that second server with that other hosting company. When I moved to the other hosting company I had this problem so I went back to the old hosting company. But now I’m on a newer server with them using a secure log in. I know what my old username and password was, but not the old server id. I hope that won’t be a problem. If so let me know because I would need to contact them to find out. Anyway, I'll try it and see what happens and let you know. .
xfrocks @rsi So you have 3 sets of ftp credentials now right? The 1st one (with the old hosting company), the 2nd one (with the new one) and the 3rd one (back with the old one). Assuming you have moved all the files yourself, you will have to run the tool 2 times. The first time with the 1st ftp credentials and once again with the 2nd set. For each set, you only need to enter one kind of information (either the host or the username or the password). The tool will look for attachments with matching information and update them.
Ok, it still didn't work... I used that update feature like you said with my old host/username/password and etc. with my new one via the Attachment menu ACP and it went through all of my attachments without any errors. The problem is when I then go to run the xenforo 'Rebuild Attachment Thumbnails' tool I get the same error below. I've replaced my old actual host/domain info with myolddomain.com to conceal it here. Unable to connect to myolddomain.com:21 Code: Error Info Thumb rebuild for #864: Unable to connect to myolddomain.com:21 - library/bdAttachmentStore/Model/File.php:315 Stack Trace #0 /home/me/public_html/library/bdAttachmentStore/Model/File.php(381): bdAttachmentStore_Model_File->_ftp_connect(Array) #1 /home/me/public_html/library/bdAttachmentStore/Model/File.php(29): bdAttachmentStore_Model_File->_ftp_saveFile(Array, '/home/me/p...', '2014/04/864_bef...', 'NC-303_ANT_INPU...') #2 /home/me/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(133): bdAttachmentStore_Model_File->saveFile('ftp', Array, '/home/me/p...', '2014/04/864_bef...', 'NC-303_ANT_INPU...') #3 /home/me/public_html/library/NFLJ/AMS/XenForo/DataWriter/AttachmentData.php(48): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/me/p...', Array, true) #4 /home/me/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(145): NFLJ_AMS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/me/p...', Array, true) #5 /home/me/public_html/library/XenForo/DataWriter/AttachmentData.php(167): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_writeAttachmentFileData('\x89PNG\r\n\x1A\n\x00\x00\x00\rIHD...', Array, true) #6 /home/me/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(55): XenForo_DataWriter_AttachmentData->_postSave() #7 /home/me/public_html/library/XenForo/DataWriter.php(1423): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave() #8 /home/me/public_html/library/XenForo/Deferred/AttachmentThumb.php(67): XenForo_DataWriter->save() #9 /home/me/public_html/library/bdAttachmentStore/XenForo/Deferred/AttachmentThumb.php(11): XenForo_Deferred_AttachmentThumb->execute(Array, Array, 7.99999809265, '') #10 /home/me/public_html/library/XenForo/Model/Deferred.php(295): bdAttachmentStore_XenForo_Deferred_AttachmentThumb->execute(Array, Array, 7.99999809265, '') #11 /home/me/public_html/library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 7.99999809265, '', false) #12 /home/me/public_html/library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, NULL, '', false) #13 /home/me/public_html/library/XenForo/ControllerAdmin/Tools.php(159): XenForo_Model_Deferred->run(true, NULL, '', false) #14 /home/me/public_html/library/XenForo/FrontController.php(351): XenForo_ControllerAdmin_Tools->actionRunDeferred() #15 /home/me/public_html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch)) #16 /home/me/public_html/admin.php(13): XenForo_FrontController->run() #17 {main} Request State Array ( [url] => http://myolddomain.com/admin.php?tools/run-deferred [_GET] => Array ( [tools/run-deferred] => ) [_POST] => Array ( [redirect] => http://myolddomain.com/admin.php?tools/rebuild&success=1 [execute] => 1 [_xfToken] => ******** ) )
Hi, I did try it one more time and it still won't work. I have just under 3000 attachments that still have my old host name attached to them, at least to the thumbnail images. Host name is was what I really meant and not my ftp user name. It was my old ftp host name which is/was my domain name. The only difference is now I am using a secure ftp host name which has the server ID, but it's still the same domain name overall. The problem is it keeps trying to rebuild the attachment thumbnails using that old host name and the 'Update Attachment Data Storage Option' tool is not updating my old host name still attached to those files. Anyway, that's the problem I've had for almost a year now. It's either due to a bug in the Attachment Store add-on or that update tool doesn't work or is just not setup to correct my actual problem. Is there anyway you can correct the problem or at least send me something that I can run that will correct it? Thanks... I really need to get this problem resolved. .
Here is one last bit of information... I did a search in my database and using the old hostname I see the total below... 2338 matches in xf_attachment_data I then did a search using my new hostname and get the total below... 1315 matches in xf_attachment_data It looks like those matches are hidden within the file hash, right? The update tool is not updating my old hostname in those 2338 matches to the new hostname and that is the problem. Anyway, to clarify more I only have those 2 hostnames associated to the attachments I have uploaded using the attachment store add-on. Those 2338 attachments were uploaded back when I was using my domain name for the actual ftp hostname. A few months ago I started using CloudFare so I had to change that ftp hostname to my server's secure hostname instead to get around the upload problem of using my domain name and that is the new 1315 attachments. That other server I was on is not even an issue because I cleaned up and redid the small handful of attachments that were uploaded when I was using it. So, it is just with the 2 different hostnames, my old one and my new one. This morning I even disabled CloudFare (paused it) and tried running the update tool again with my domain hostname working and it still did not work. The database search still yields the 2338 matches in the xf_attachment_data table with the old hostname. I would gladly make a donation if you can correct this problem because I'm not the only one having it. That update tool is just not working to correct the problem. Either that or if you can provide me with a script I can run to update the hostname in the 2338 attachments. Just send me a PM if necessary and if a donation is required. Thank You .
Actually the tool keep the old value for tracking reason so it looks like it successfully updated config for 1315 files but failed for 1023 (2338 minus 1315). It would be useful if you can get a dump of two rows for me: one that matches the old host without the new host and one that matches both old host/new host.
Hi, No, the 1315 attachments were uploaded using my current hostname so those attachments are fine. I should have made that more clear I guess, sorry... It is the 2338 attachments that need the hostname updated to my current hostname. The ACP tool didn't need to do anything to the 1315 attachments. So then I guess that is the problem if it is keeping the old hostname for tracking, correct? How can I update those 2338 attachments to my current hostname? If you could send me something or update the tool where I can just enter the hostname and other information if necessary then I could just run it. If you still need more information let me know. Thanks... .
xfrocks @rsi When I said the tool keep old host for tracking, it still update the attachment to use the new host but then store the old host separately. That explains why you will still find those record using the old host even for a successful update. Anyway, a dump of the failed attachment would be useful if possible?
rsi @xfrocks Hi, The actual update tool is not creating any errors, it goes through all of the attachments and then finishes like normal. It's when I go to run the built-in xenforo 'Rebuild Attachment Thumbnails' tool is when I get the error below. I have to start it and then stop it real quick just so I can save the error below before the hosting company locks me out due to all of the failed connection attempts. That is the problem, if I try to run it they lock me out because it triggers a security event so it’s hard for me to just let it run because I know what will happen. I could try it, but I’m not sure it would even produce any dump. Where would I look for it? In my root directory? I get the same error below for each of the 2338 attachments and it says “Unable to connect to myolddomain.com:21” myolddomain.com is my old hostname connected to those 2338 attachments. Code: Error Info Thumb rebuild for #864: Unable to connect to myolddomain.com:21 - library/bdAttachmentStore/Model/File.php:315 Stack Trace #0 /home/me/public_html/library/bdAttachmentStore/Model/File.php(381): bdAttachmentStore_Model_File->_ftp_connect(Array) #1 /home/me/public_html/library/bdAttachmentStore/Model/File.php(29): bdAttachmentStore_Model_File->_ftp_saveFile(Array, '/home/me/p...', '2014/04/864_bef...', 'NC-303_ANT_INPU...') #2 /home/me/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(133): bdAttachmentStore_Model_File->saveFile('ftp', Array, '/home/me/p...', '2014/04/864_bef...', 'NC-303_ANT_INPU...') #3 /home/me/public_html/library/NFLJ/AMS/XenForo/DataWriter/AttachmentData.php(48): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/me/p...', Array, true) #4 /home/me/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(145): NFLJ_AMS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/me/p...', Array, true) #5 /home/me/public_html/library/XenForo/DataWriter/AttachmentData.php(167): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_writeAttachmentFileData('\x89PNG\r\n\x1A\n\x00\x00\x00\rIHD...', Array, true) #6 /home/me/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(55): XenForo_DataWriter_AttachmentData->_postSave() #7 /home/me/public_html/library/XenForo/DataWriter.php(1423): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave() #8 /home/me/public_html/library/XenForo/Deferred/AttachmentThumb.php(67): XenForo_DataWriter->save() #9 /home/me/public_html/library/bdAttachmentStore/XenForo/Deferred/AttachmentThumb.php(11): XenForo_Deferred_AttachmentThumb->execute(Array, Array, 7.99999809265, '') #10 /home/me/public_html/library/XenForo/Model/Deferred.php(295): bdAttachmentStore_XenForo_Deferred_AttachmentThumb->execute(Array, Array, 7.99999809265, '') #11 /home/me/public_html/library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 7.99999809265, '', false) #12 /home/me/public_html/library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, NULL, '', false) #13 /home/me/public_html/library/XenForo/ControllerAdmin/Tools.php(159): XenForo_Model_Deferred->run(true, NULL, '', false) #14 /home/me/public_html/library/XenForo/FrontController.php(351): XenForo_ControllerAdmin_Tools->actionRunDeferred() #15 /home/me/public_html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch)) #16 /home/me/public_html/admin.php(13): XenForo_FrontController->run() #17 {main} Request State Array ( [url] => http://myolddomain.com/admin.php?tools/run-deferred [_GET] => Array ( [tools/run-deferred] => ) [_POST] => Array ( [redirect] => http://myolddomain.com/admin.php?tools/rebuild&success=1 [execute] => 1 [_xfToken] => ******** ) ) .
I went ahead and tried to run the xenforo 'Rebuild Attachment Thumbnails' tool again and it pretty much wants to freeze when it tries to rebuild those 2338 attachments because there is a very long delay when it tries to connect to each one of them so I had to stop it. It would literally take forever and the hosting company will lock out my ip address because of the repeated connection attempts so I am unable to get a dump. Anyway, it still looks like it’s trying to use my old hostname to rebuild those 2338 attachments or less because not all of them are images and it produces the same error as in my last post. It rebuilds the other images ok until it gets to the ones with the old hostname. At least that’s what it looks like. .
rsi @xfrocks I just sent it to you via a conversation message. Hopefully I exported it correctly using phpMyAdmin. If not let me know. .
Hi, I finally made the update tool work. I had to uncheck the Maintain File Name and the Enforce Attachment Permission via Home>Options>Attachments, save it, then I ran the tool again. Then I went back and rechecked those boxes, saved it, and then ran the tool again. I don’t know which step made it work or which of the two checkboxes was the problem, but if I had to guess it's when I unchecked the Enforce Attachment Permission box and saved it and ran the tool the first time, but I could be wrong. In any event there seems to be some conflict with one of those two options or maybe both making the update tool not completely work. Those attachments I had though were originally uploaded with the Maintain File Name off so it could be that one as well. I’m wondering if you need to add those two checkbox options to the update tool? .
xfrocks @rsi Interesting find. We will go through it again to find out what exactly happened. Of course, those checkboxes shouldn't affect the tool...
rsi @xfrocks One or both of those two checkboxes were somehow locking or preventing those SQL entries from being updated. That why it wasn't updating the hostname. I may or may not have uploaded those old attachments with one or both those boxes unchecked, but I think it was in fact a mixture. Some where and some were not. In any event, unchecking those boxes and running the tool and then rechecking them and doing it again is what unlocked them. I'm not sure if it was after the first step or the second step, but that it what did it. .
One last thing, the error below still pops up when doing the xenforo file heatlh check. I think this has already been brought up before, but when you release the next update you might want to correct it. I think also adding the Maintain File Name and the Enforce Attachment Permission checkboxes to the update tool may be a good idea as well because it was an issue, at least in my case it was. Thanks for you help... .
xfrocks @vktechnology It takes some time before the files are actually deleted. Have you looked again? They should be deleted by now.
Hi all files are deleted now after 3 hours xx minutes My config is Store file in External Data directory I want to test again If i want to delete faster than 3 hours which file should i run ?
Hi I got answer now how to delete fast After delete I go to run "Hourly Clean Up" and all files deleted
Hi @xfrocks What I have to do to move large attachments ? 1) I want to move 3 millions attachments about 2 TB size 2) I use this option "Store file in External Data directory" 3) Attachments on same HDD Do I have to just click Rebuild Now button ? on Rebuild Caches > Move Attachment Data Thanks
xfrocks @vktechnology That's a lot of files, unfortunately there is no shortcut to do it quick since the files structure is changed and must be done together with db update to make sure it works correctly. The steps you stated are right, it should run quite fast because the add-on will do file moving (renaming) instead of copying.
xfrocks updated [bd] Attachment Store with a new update entry: Minor Update Read the rest of this update entry...
