This is in the FAQ: Q: Can I use Digital Ocean Space? A: Yes. The add-on supports all S3-compatible storage services. For DO, put something like this in your config.php: Code: $config['bdAttachmentStore_s3Buckets'] = [ 'my-do-space' => [ 'key' => 'ABC123', 'secret' => 'xyz456', 'endpoint' => 'https://my-do-space.nyc3.digitaloceanspaces.com', 'url' => 'https://nyc3.digitaloceanspaces.com/my-do-space', ], ]; You will need to replace your DO Space name, key and secret for it to work. After updating config.php, enter "my-do-space" as the bucket name for S3 in XenForo AdminCP options page. That seems clear. But what settings do I enter in /admin.php?options/list/attachments ? Store file in External Data directory? OR Store file in Amazon S3? Do I need to set a Root URL? Do I need to set externalDataUrl as well? In digitalOcean I can set Advanced CORS Options to allow certain methods: Get, Put, Delete, Post, Head. Do I need to set these?
xfrocks @alfa1 In AdminCP, select S3 mode and enter the bucket name "my-do-space" as you put in the config.php file. Everything else can be left empty. And you don't need the CORS options.
Im getting this error: Zend_Http_Client_Adapter_Exception: Unable to Connect to ssl://myforumm.ams3.digitaloceanspaces.com:443. Error #110: Connection timed out - library/Zend/Http/Client/Adapter/Socket.php:235 Generated By: Alfa, A moment ago Stack Trace #0 /library/Zend/Http/Client.php(973): Zend_Http_Client_Adapter_Socket->connect('ssl://myforum...', 443, true) #1 /library/bdAttachmentStore/ShippableHelper/S3.php(98): Zend_Http_Client->request('PUT') #2 /library/Zend/Service/Amazon/S3.php(412): bdAttachmentStore_ShippableHelper_S3->_makeRequest('PUT', 'myforumm/2005...', Array, Array, 'BM\x8A!\x03\x00\x00\x00\x00\x006\x00\x00\x00(...') #3 /library/Zend/Service/Amazon/S3.php(454): Zend_Service_Amazon_S3->putObject('myforumm/2005...', 'BM\x8A!\x03\x00\x00\x00\x00\x006\x00\x00\x00(...', Array) #4 /library/bdAttachmentStore/Model/File.php(185): Zend_Service_Amazon_S3->putFile('/home/myforum...', 'myforumm/2005...', Array) #5 /library/bdAttachmentStore/Model/File.php(27): bdAttachmentStore_Model_File->_s3_saveFile(Array, '/home/myforum...', '2005/12/258529_...', 'injury.b...') #6 /library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(133): bdAttachmentStore_Model_File->saveFile('s3', Array, '/home/myforum...', '2005/12/258529_...', 'methlabinjury.b...') #7 /library/NFLJ/UBS/XenForo/DataWriter/AttachmentData.php(48): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/myforum...', Array, false) #8 /library/NFLJ/RMS/XenForo/DataWriter/AttachmentData.php(48): NFLJ_UBS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/myforum...', Array, false) #9 /library/NFLJ/AMS/XenForo/DataWriter/AttachmentData.php(48): NFLJ_RMS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/myforum...', Array, false) #10 /library/XenGallery/DataWriter/AttachmentData.php(53): NFLJ_AMS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/myforum...', Array, false) #11 /library/XenForo/DataWriter/AttachmentData.php(144): XenGallery_DataWriter_AttachmentData->_writeAttachmentFile('/home/myforum...', Array) #12 /library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(55): XenForo_DataWriter_AttachmentData->_postSave() #13 /library/XenForo/DataWriter.php(1423): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave() #14 /library/bdAttachmentStore/Helper/Mover.php(61): XenForo_DataWriter->save() #15 /library/bdAttachmentStore/CacheRebuilder/AttachmentData.php(53): bdAttachmentStore_Helper_Mover::move(Array, Object(XFA_ExtendedCover_Extends_XenForo_Model_Attachment), 's3', Object(bdAttachmentStore_Model_File)) #16 /library/XenForo/ControllerHelper/CacheRebuild.php(26): bdAttachmentStore_CacheRebuilder_AttachmentData->rebuild(6, Array, NULL) #17 /library/XenForo/ControllerAdmin/Tools.php(81): XenForo_ControllerHelper_CacheRebuild->rebuildCache(Array, 'https://my-f...', 'admin.php?tools...', true) #18 /library/XenForo/FrontController.php(369): XenForo_ControllerAdmin_Tools->actionCacheRebuild() #19 /library/XenForo/FrontController.php(152): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch)) #20 /admin.php(13): XenForo_FrontController->run() #21 {main} Request State array(3) { ["url"] => string(53) "https://my-forum.com/admin.php?tools/cache-rebuild" ["_GET"] => array(1) { ["tools/cache-rebuild"] => string(0) "" } ["_POST"] => array(5) { ["process"] => string(1) "1" ["caches"] => string(53) "[["bdAttachmentStore_AttachmentData",{"batch":"50"}]]" ["position"] => string(1) "0" ["redirect"] => string(47) "https://my-forum.com/admin.php?tools/rebuild" ["_xfToken"] => string(8) "********" } }
I am moving about 26GB of attachments to a DO space. Its doing about 60MB/hour which would take 18 days if it would keep at it non stop. Is this normal?
xfrocks @alfa1 For large move, we normally recommend using external uploader + running the update storage options tool instead of running the move tool directly.
Am I understanding correctly that bd attachment changes the folder structure and file names? I see thumbs & full images in the same droplet folders, while XF has thumbs in a separate folder. XF: /data/attachment-files/2007/01/thumb/ID_thumbfile.jpg DO Droplet: 2007/01/ID_thumbfile.jpg The ID on default XF is different from the DO droplet. How can I upload to the files in the new format?
Correct. By default, XenForo keeps thumbnails in data directory and full size in internal_data. It's expected to have different ID between XenForo and DO. To move large amount of files, you can switch to external data storage. Run the move tool to copy files from internal_data to data. Then upload the data/attachment-files directory using external uploader. Finally run the update storage options tool to update the database to use DO instead of the data directory.
alfa1 @xfrocks Am I correct that in this situation it will use the same folder structure and ID's as XF? Do I upload to /data/attachment-files/ What do I need to put in the config.php? Also: Does the Data Storage addon work also with Digital Ocean Spaces like bd attachment store does? (external data storage option) As you know I have quite a large amount of data to transfer to the Spaces, so I will need to use an external uploader (CyberDuck) for that.
alfa1 @xfrocks It seems too big to complete. I still have some attachments in internal data. To move that the move tool needs to go through a very large amount of internal data. I do not get passed 26k.
xfrocks @alfa1 The tool is designed to run for a long time but it will eventually move everything. How long have you let it running?
I'm not immediately planning to upgrade to XF2, but in preparation I'd like to move my attachments back to default. I've been using this particular addon for a few years, and users upload 50-100 new images per day. What should I consider before running the "rebuild" ?
xfrocks @motowebmaster You don't need to move back to default to prepare for XF2 upgrade. This add-on has a version that works with XF2 too, just upgrade XenForo, upgrade add-on and you are done.
To explain: bd attachment store has been installed for a long time and attachment files are served from /data/attachment-files/ The DO droplet has been synched so that the folder mydroplet.ams3.digitaloceanspaces.com/data/attachment-files/ contains the same files. 'Update Attachment Data Storage Options' was run to 'Store file in External Data directory' When clicking an attachment it tries to open gives the error 'no such key': Code: This XML-file does not appear to have an associated style information. Please see below for document structure. <Error><Code>NoSuchKey</Code><BucketName>myspace</BucketName><RequestId>xxx-ams3a</RequestId><HostId>xxx-ams3a-ams3</HostId></Error>
xfrocks @alfa1 Have you try uploading new attachment? If it get uploaded, that means the DO space has been correctly configured. If it can't be uploaded, that means the config is not correct, that may explain the error in step 4.
Days. It keeps hitting an error: Zend_Http_Client_Adapter_Exception: Unable to connect to HTTPS proxy. Server response: - library/Zend/Http/Client/Adapter/Proxy.php:241 Generated By: Alfa, Today at 11:55 AM Stack Trace #0 /library/Zend/Http/Client/Adapter/Proxy.php(147): Zend_Http_Client_Adapter_Proxy->connectHandshake('ams3.digitaloce...', 443, '1.1', Array) #1 /library/Zend/Http/Client.php(987): Zend_Http_Client_Adapter_Proxy->write('PUT', Object(Zend_Uri_Http), '1.1', Array, '\xFF\xD8\xFF\xE0\x00\x10JFIF\x00\x01\x01\x01\x00...') #2 /library/bdAttachmentStore/ShippableHelper/S3.php(113): Zend_Http_Client->request('PUT') #3 /library/Zend/Service/Amazon/S3.php(412): bdAttachmentStore_ShippableHelper_S3->_makeRequest('PUT', 'drugsforum/data...', Array, Array, '\xFF\xD8\xFF\xE0\x00\x10JFIF\x00\x01\x01\x01\x00...') #4 /library/Zend/Service/Amazon/S3.php(454): Zend_Service_Amazon_S3->putObject('drugsforum/data...', '\xFF\xD8\xFF\xE0\x00\x10JFIF\x00\x01\x01\x01\x00...', Array) #5 /library/bdAttachmentStore/Model/File.php(192): Zend_Service_Amazon_S3->putFile('/home/drugsforu...', 'drugsforum/data...', Array) #6 /library/bdAttachmentStore/Model/File.php(31): bdAttachmentStore_Model_File->_s3_saveFile(Array, '/home/drugsforu...', '2012/08/281000_...', 'Xanax, dosage u...') #7 /library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(132): bdAttachmentStore_Model_File->saveFile('s3', Array, '/home/drugsforu...', '2012/08/281000_...', 'Xanax, dosage u...') #8 /library/NFLJ/UBS/XenForo/DataWriter/AttachmentData.php(48): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/drugsforu...', Array, true) #9 /library/NFLJ/RMS/XenForo/DataWriter/AttachmentData.php(48): NFLJ_UBS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/drugsforu...', Array, true) #10 /library/NFLJ/AMS/XenForo/DataWriter/AttachmentData.php(48): NFLJ_RMS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/drugsforu...', Array, true) #11 /library/XenGallery/DataWriter/AttachmentData.php(53): NFLJ_AMS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/drugsforu...', Array, true) #12 /library/XenForo/DataWriter/AttachmentData.php(160): XenGallery_DataWriter_AttachmentData->_writeAttachmentFile('/home/drugsforu...', Array, true) #13 /library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(54): XenForo_DataWriter_AttachmentData->_postSave() #14 /library/XenForo/DataWriter.php(1423): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave() #15 /library/bdAttachmentStore/Helper/Mover.php(95): XenForo_DataWriter->save() #16 /library/bdAttachmentStore/Helper/Mover.php(52): bdAttachmentStore_Helper_Mover::saveDw(Array, Array, '/home/drugsforu...', '/home/drugsforu...') #17 /library/bdAttachmentStore/Deferred/AttachmentData.php(46): bdAttachmentStore_Helper_Mover::move(Array, Object(bdAttachmentStore_XenForo_Model_Attachment), 's3', Object(bdAttachmentStore_Model_File)) #18 /library/XenForo/Model/Deferred.php(295): bdAttachmentStore_Deferred_AttachmentData->execute(Array, Array, 7.99999690056, '') #19 /library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 7.99999690056, '', false) #20 /library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, 8, '', false) #21 /library/XenForo/ControllerAdmin/Tools.php(159): XenForo_Model_Deferred->run(true, NULL, '', false) #22 /library/XenForo/FrontController.php(369): XenForo_ControllerAdmin_Tools->actionRunDeferred() #23 /library/XenForo/FrontController.php(152): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch)) #24 /admin.php(13): XenForo_FrontController->run() #25 {main} Request State array(3) { ["url"] => string(47) "https://xxx/admin.php?tools/run-deferred" ["_GET"] => array(1) { ["tools/run-deferred"] => string(0) "" } ["_POST"] => array(3) { ["redirect"] => string(52) "https://xxx/admin.php?tools/rebuild&success=1" ["execute"] => string(1) "1" ["_xfToken"] => string(8) "********" } }
xfrocks @alfa1 It said it was not able to connect to the proxy, is your proxy up and running? Is it able to handle lots of connections continuously?
xfrocks @alfa1 By the way, it is recommended to use the tool to move files from internal data to external data only. You are moving files to S3 which will be slow, considering you have GBs of files.
alfa1 @xfrocks I am now doing that. But unfortunately after about 25k it is suddenly finished. There are hundreds of thousands. I can find some that are not done.
xfrocks @alfa1 You are moving files from internal_data -> external data and it stuck at 25k? Do you have enough disk space on the server? During move, it may take twice as much space because the files are copied over before being deleted.
There is plenty of diskspace. Maybe this is related? ErrorException: Fatal Error: Uncaught Error: Access to undeclared static property: bdAttachmentStore_ShippableHelper_TempFile::$_maxDownloadSize in /library/bdAttachmentStore/ShippableHelper/TempFile.php:194 Stack trace: #0 [internal function]: bdAttachmentStore_ShippableHelper_TempFile::download_curlProgressFunction(Resource id #26780, 0, 0, 0, 0) #1 {main} thrown - library/bdAttachmentStore/ShippableHelper/TempFile.php:194 Generated By: Unknown Account, Today at 1:29 AM Stack Trace #0 [internal function]: XenForo_Application::handleFatalError() #1 {main} Request State array(3) { ["url"] => string(113) "https://forum.com/forums/caffeine-c...2&order=post_date&direction=asc&prefix_id=196" ["_GET"] => array(4) { ["prefix_ids"] => array(1) { [0] => string(3) "192" } ["order"] => string(9) "post_date" ["direction"] => string(3) "asc" ["prefix_id"] => string(3) "196" } ["_POST"] => array(0) { } }
xfrocks @alfa1 That error should never happen. Can you copy and paste the contents of "library/bdAttachmentStore/ShippableHelper/TempFile.php" to verify?
I am currently running attachments from DO spaces. However, something has gone wrong with the conversion from default location to external data and quite a lot were skipped. So the synch from external data to spaces also misses these files. What is the way to fix this? I assume that I need to use the move tool to move from internal data to external data. But how does the script know where to move from? i.e. if I now set my options to use external data and then use the move tool, then will it move from spaces to external data or from internal data to external data?
xfrocks @alfa1 Do you know why "quite a lot were skipped"? You can just keep the DO Space options and run the tools again. It will find the existing files in internal data and upload straight to DO. It's recommended to move first to external data but that step is not a hard requirement. The move tool is capable of moving files across all kind of storage, it is just a bit slow. Which should be fine if you only have a handful of files left. However, you need to find out why those files have not been moved already. If there are bad files (non existent, corrupted, etc.) in internal data, of course the move tool will skip it.
alfa1 @xfrocks I assume because of time outs. It never completes and there is no way to resume where it left off. I find hundreds of mentions of broken DO Space links in google webmaster tools. Its not a handful. Its thousands.
xfrocks @alfa1 Moving from internal_data -> external data is a local operation and it should never "time out" I think. You probably should get 3-5 links from GWT and cross check to make sure: 1 the attachment itself is not deleted (sometimes attachments are deleted) and 2 the file were left in internal_data.
I may have found the answer to my question. Using the move tool seems to delete from the spaces: Code: Zend_Http_Client_Exception: Unable to read response, or response is empty - library/Zend/Http/Client.php:993 Generated By: Unknown Account, Today at 7:10 PM Stack Trace #0 /library/bdAttachmentStore/ShippableHelper/S3.php(113): Zend_Http_Client->request('DELETE') #1 /library/Zend/Service/Amazon/S3.php(500): bdAttachmentStore_ShippableHelper_S3->_makeRequest('DELETE', 'spaces/data...') #2 /library/bdAttachmentStore/Model/File.php(210): Zend_Service_Amazon_S3->removeObject(' spaces/data...') #3 /library/bdAttachmentStore/Model/File.php(45): bdAttachmentStore_Model_File->_s3_deleteFile(Array, '2009/07/296586_...') #4 /library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(76): bdAttachmentStore_Model_File->deleteFile('s3', Array, '2009/07/296586_...') #5 /library/XenForo/DataWriter.php(1793): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postDelete() #6 /library/XenForo/Model/Attachment.php(678): XenForo_DataWriter->delete() #7 /library/bdAttachmentStore/XenForo/Model/Attachment.php(71): XenForo_Model_Attachment->deleteUnusedAttachmentData() #8 /library/XenForo/CronEntry/CleanUp.php(18): bdAttachmentStore_XenForo_Model_Attachment->deleteUnusedAttachmentData() #9 /library/XenForo/Model/Cron.php(356): XenForo_CronEntry_CleanUp::runHourlyCleanUp(Array) #10 /library/XenForo/Deferred/Cron.php(24): XenForo_Model_Cron->runEntry(Array) #11 /library/XenForo/Model/Deferred.php(295): XenForo_Deferred_Cron->execute(Array, Array, 7.9999990463257, '') #12 /library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 7.9999990463257, '', false) #13 /library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, 8, '', false) #14 /deferred.php(23): XenForo_Model_Deferred->run(false) #15 {main} Request State array(3) { ["url"] => string(36) "https://forum.com/deferred.php" ["_GET"] => array(0) { } ["_POST"] => array(3) { ["_xfRequestUri"] => string(83) "/threads/173636/" ["_xfNoRedirect"] => string(1) "1" ["_xfResponseType"] => string(4) "json" } } So what do I do now?
xfrocks @alfa1 The error log captured is about the cron job. It was trying to delete a file from DO (because the attachment has been deleted) but couldn't do it. I also notice an extra space in this line: Code: #2 /library/bdAttachmentStore/Model/File.php(210): Zend_Service_Amazon_S3->removeObject(' spaces/data...') Probably a bad config?
I am encountering this error when running the move tool: Code: Zend_Http_Client_Adapter_Exception: Read timed out after 10 seconds - library/Zend/Http/Client/Adapter/Socket.php:514 Generated By: Alfa, 5 minutes ago Stack Trace #0 /library/Zend/Http/Client/Adapter/Socket.php(330): Zend_Http_Client_Adapter_Socket->_checkSocketReadTimeout() #1 /library/Zend/Http/Client/Adapter/Socket.php(335): Zend_Http_Client_Adapter_Socket->read() #2 /library/Zend/Http/Client.php(989): Zend_Http_Client_Adapter_Socket->read() #3 /library/bdAttachmentStore/ShippableHelper/S3.php(113): Zend_Http_Client->request('PUT') #4 /library/Zend/Service/Amazon/S3.php(412): bdAttachmentStore_ShippableHelper_S3->_makeRequest('PUT', ‘myspaces/data...', Array, Array, '\x89PNG\r\n\x1A\n\x00\x00\x00\rIHD...') #5 /library/Zend/Service/Amazon/S3.php(454): Zend_Service_Amazon_S3->putObject(‘myspaces/data...', '\x89PNG\r\n\x1A\n\x00\x00\x00\rIHD...', Array) #6 /library/bdAttachmentStore/Model/File.php(192): Zend_Service_Amazon_S3->putFile('/home/nginx/dom...', ‘myspaces/data...', Array) #7 /library/bdAttachmentStore/Model/File.php(31): bdAttachmentStore_Model_File->_s3_saveFile(Array, '/home/nginx/dom...', '2009/07/298272_...', '200px-Dimethylc...') #8 /library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(132): bdAttachmentStore_Model_File->saveFile('s3', Array, '/home/nginx/dom...', '2009/07/298272_...', '200px-Dimethylc...') #9 /library/NFLJ/UBS/XenForo/DataWriter/AttachmentData.php(48): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/nginx/dom...', Array, false) #10 /library/NFLJ/RMS/XenForo/DataWriter/AttachmentData.php(48): NFLJ_UBS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/nginx/dom...', Array, false) #11 /library/NFLJ/AMS/XenForo/DataWriter/AttachmentData.php(48): NFLJ_RMS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/nginx/dom...', Array, false) #12 /library/XenGallery/DataWriter/AttachmentData.php(53): NFLJ_AMS_XenForo_DataWriter_AttachmentData->_writeAttachmentFile('/home/nginx/dom...', Array, false) #13 /library/XenForo/DataWriter/AttachmentData.php(144): XenGallery_DataWriter_AttachmentData->_writeAttachmentFile('/home/nginx/dom...', Array) #14 /library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(54): XenForo_DataWriter_AttachmentData->_postSave() #15 /library/XenForo/DataWriter.php(1423): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave() #16 /library/bdAttachmentStore/Helper/Mover.php(95): XenForo_DataWriter->save() #17 /library/bdAttachmentStore/Helper/Mover.php(52): bdAttachmentStore_Helper_Mover::saveDw(Array, Array, '/home/nginx/dom...', '/home/nginx/dom...') #18 /library/bdAttachmentStore/Deferred/AttachmentData.php(46): bdAttachmentStore_Helper_Mover::move(Array, Object(bdAttachmentStore_XenForo_Model_Attachment), 's3', Object(bdAttachmentStore_Model_File)) #19 /library/XenForo/Model/Deferred.php(295): bdAttachmentStore_Deferred_AttachmentData->execute(Array, Array, 8, '') #20 /library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 8, '', false) #21 /library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, 8, '', false) #22 /library/XenForo/ControllerAdmin/Tools.php(159): XenForo_Model_Deferred->run(true, NULL, '', false) #23 /library/XenForo/FrontController.php(369): XenForo_ControllerAdmin_Tools->actionRunDeferred() #24 /library/XenForo/FrontController.php(152): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch)) #25 /admin.php(13): XenForo_FrontController->run() #26 {main} Request State array(3) { ["url"] => string(52) "https://forum.com/admin.php?tools/run-deferred" ["_GET"] => array(1) { ["tools/run-deferred"] => string(0) "" } ["_POST"] => array(3) { ["redirect"] => string(57) "https://forum.com/admin.php?tools/rebuild&success=1" ["execute"] => string(1) "1" ["_xfToken"] => string(8) "********" } } I am processing 5 at a time.
xfrocks @alfa1 This is a low level network error, you probably need to check your connection. Especially the outgoing direction (from server to the internet).
@xfrocks - the server is with DO, and going to DO Spaces. Using S3CMD works fine for moving data over to DO Spaces.
Attachments are set to default location and I am getting these errors: XenForo_Exception: S3 key cannot be empty - library/bdAttachmentStore/Model/File.php:275 Generated By: Unknown Account, Today at 7:10 AM Stack Trace #0 /library/bdAttachmentStore/Model/File.php(208): bdAttachmentStore_Model_File->_s3_getS3Connection(Array) #1 /library/bdAttachmentStore/Model/File.php(45): bdAttachmentStore_Model_File->_s3_deleteFile(Array, '2007/12/261819_...') #2 /library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(76): bdAttachmentStore_Model_File->deleteFile('s3', Array, '2007/12/261819_...') #3 /library/XenForo/DataWriter.php(1793): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postDelete() #4 /library/XenForo/Model/Attachment.php(678): XenForo_DataWriter->delete() #5 /library/bdAttachmentStore/XenForo/Model/Attachment.php(71): XenForo_Model_Attachment->deleteUnusedAttachmentData() #6 /library/XenForo/CronEntry/CleanUp.php(18): bdAttachmentStore_XenForo_Model_Attachment->deleteUnusedAttachmentData() #7 /library/XenForo/Model/Cron.php(356): XenForo_CronEntry_CleanUp::runHourlyCleanUp(Array) #8 /library/XenForo/Deferred/Cron.php(24): XenForo_Model_Cron->runEntry(Array) #9 /library/XenForo/Model/Deferred.php(295): XenForo_Deferred_Cron->execute(Array, Array, 7.9999990463257, '') #10 /library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 7.9999990463257, '', false) #11 /library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, 8, '', false) #12 /deferred.php(23): XenForo_Model_Deferred->run(false) #13 {main} Request State array(3) { ["url"] => string(36) "https://my-forum.com/deferred.php" ["_GET"] => array(0) { } ["_POST"] => array(3) { ["_xfRequestUri"] => string(33) "/threads/2161/" ["_xfNoRedirect"] => string(1) "1" ["_xfResponseType"] => string(4) "json" } } bd attachments should not be trying to access S3. Im not sure what its trying to do but it should either affect default location or move from external data to default.
xfrocks @alfa1 The error quoted happened because an attachment has been deleted. The file for that attachment was uploaded to S3 so the add-on was trying to delete the associated file from S3.
alfa1 @xfrocks The attachment does not exist on S3. it was not uploaded to S3. But why is it looking for the attachment on S3 in the first place when no S3 is used anymore?
xfrocks @alfa1 So there was a period of time when some of the files are in S3? My point is: if the database indicates an attachment file is in S3, the add-on will clean it up when the attachment gets deleted.
For example: When I click on a certain attachment, the link goes to https://mybucket.s3.amazonaws.com/2007/08/261191_3719d8bebbca852954785bfb91b98aa9.pdf How do I get it to link to /internal_data/attachments/etc ?
@xfrocks I am still very much stuck with tens of thousands of dead links. Please help. I want to get back to the default situation without s3 or external data. But the 'Update Attachment Data Storage Options' tool doesn't seem to work for this. Attachments still keep pointing to S3 / DO droplet. I also tried the move tool. That will just delete the attachments without adding those to the data/attachments folder. How do I fix this?
TheComputerGuy @alfa1 I am/was stuck in the same situation. Really killing community spirits to lose so many attachments. Sorry to hear you're having the same trouble.
xfrocks @TheComputerGuy Hmm, what is your issue? I don't recall you have posted anything about this before?
@xfrocks Google is hitting my site really hard due to 140K dead attachment links. Please help. One issue seems to be that the entry in the table attachment_data has the wrong bd_attachmentstore_engine with related bd_attachmentstore_options blob For example: Entry: 261191 bd_attachmentstore_engine: s3 bd_attachmentstore_options blob: Code: a:5:{s:6:"bucket";s:10:"mybucket";s:3:"key";s:0:"";s:6:"secret";s:0:"";s:6:"region";s:9:"us-east-1";s:8:"protocol";s:4:"auto";} The file is not on s3 but here: /data/attachment-files/2007/08/127677_filename.pdf There are 13K entries that have s3 listed. There are 350 entries that have external_data listed. Can I perform some database queries to fix this? Then there is another issue: This has 260K files: /data/attachment-files/ This has 119K files: /internal_data/attachments/ This has 108K files: /data/attachments/ Tables: xf_attachment has 134K entries. attachment_data has 140K entries. Do you have any idea how to fix this?
maxicep @alfa1 Did you have set your current amazon s3 bucket as public? Because If bucket is public, you should call your file directly.
@alfa1 didn't we talk and fix this issue already? No you can't, the data should be updated via the storage option update tool. Can you document in full what steps that you have done? You have files in S3, they were working then what? You copied them from S3 manually and the links broke?
alfa1 @xfrocks We didnt talk to fix this issue. We did talk and fix another issue with getting my proxy working earlier this year. Here is the chain of events: default location: /internal_data/attachments/ moved files to /data/attachment_files/ with move tool. moved files to DO spaces with move tool, but failed moved files to DO spaces with CLI synch moved files back to /data/attachment_files/ with CLI synch moved files back to default location with move tools, but failed. Instead move tool only deletes files.
alfa1 @xfrocks Please contact me to resolve this mess. I still have a mass of dead links and failing attachments.
McAtze @alfa1 Me too. With add-on all thumbnails are gone, after uninstall all attachments are gone..
@xfrocks I installed this add-on today and try to use it with XFMG 1.1.17. But after install, setting up and rebuild attachment data, all my thumbnails from XFMG is gone. Rebuild thumbnails doesn't work. I asked ChrisD to (https://xenforo.com/community/threads/no-thumbnail-for-uploaded-videos.156084/post-1295979), but he says i should ask you. What can i do?
