[HOWTO] XenForo attachments via Amazon S3 and CloudFront

Discussion in 'Other' started by xfrocks, May 3, 2013.

  1. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    Try disable delay processing for now and upload files again. And send me the URL if possible. Sounds like a add-on conflict.
     
  2. Russoroni

    Russoroni New Member

    my amazon bucket url is attachments.trailpeople.org - attachments.trailpeople.org.s3.amazonaws.com
     
  3. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    Please start a conversation with me with the thread URL not attachment's :D Also, an user account would be useful.
     
  4. 3rd Angle

    3rd Angle New Member

    #38 3rd Angle, Oct 30, 2014
    Last edited: Oct 30, 2014
    hi @xfrocks

    I have purchased and installed this addon.

    I have rebuilt the cache and used this option Update Attachment Data Storage Options to store in my S3 bucket.. it ran without any problem but when i checked in my s3 bucket in AWS console it is showing as empty.....

    WHat is the best way to check if this is working or not...?

    I uploaded a picture in my XenMediaGallery and got this error

    This XML file does not appear to have any style information associated with it. The document tree is shown below.
    <Error>
    <Code>AccessDenied</Code>
    <Message>Access denied</Message>
    </Error>
     
  5. 3rd Angle

    3rd Angle New Member

    [Update] Now i can upload image in xenmediagallery and it is stored in S3/served via cloudfront...


    But my basic requirement is not met yet,.... I am trying to off load the existing attachment to s3 and delete those attachment from my server...

    What do i have to do?
     
  6. 3rd Angle

    3rd Angle New Member

    ok just found out i should be using move attachment data...


    well.... i get this error when i try to move attachment data

    Code:
    Server Error
    
    Cannot read file /home/hippy/public_html/mydomain.com/data/attachments/0/225-34da4cccca52686fbe3f605dc9e5f597.jpg
    
    Zend_Service_Amazon_S3->putFile() in bdAttachmentStore/Model/File.php at line 164
    bdAttachmentStore_Model_File->_s3_saveFile() in bdAttachmentStore/Model/File.php at line 28
    bdAttachmentStore_Model_File->saveFile() in bdAttachmentStore/XenForo/DataWriter/AttachmentData.php at line 145
    bdAttachmentStore_XenForo_DataWriter_AttachmentData->_writeAttachmentFile() in XenGallery/DataWriter/AttachmentData.php at line 62
    XenGallery_DataWriter_AttachmentData->_writeAttachmentFile() in XenForo/DataWriter/AttachmentData.php at line 159
    XenForo_DataWriter_AttachmentData->_postSave() in bdAttachmentStore/XenForo/DataWriter/AttachmentData.php at line 63
    bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave() in XenForo/DataWriter.php at line 1397
    XenForo_DataWriter->save() in bdAttachmentStore/Helper/Mover.php at line 65
    bdAttachmentStore_Helper_Mover::move() in bdAttachmentStore/CacheRebuilder/AttachmentData.php at line 55
    bdAttachmentStore_CacheRebuilder_AttachmentData->rebuild() in XenForo/ControllerHelper/CacheRebuild.php at line 26
    XenForo_ControllerHelper_CacheRebuild->rebuildCache() in XenForo/ControllerAdmin/Tools.php at line 82
    XenForo_ControllerAdmin_Tools->actionCacheRebuild() in XenForo/FrontController.php at line 337
    XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
    XenForo_FrontController->run() in /home/hippy/public_html/mydomain.com/admin.php at line 13
     
  7. 3rd Angle

    3rd Angle New Member

    Hi @xfrocks

    One of the issue that has been identified by @MattW is that some of the attachments are not present due to deletion. Can you help me bypass all the deleted attachments and move those which exists presently... Since the lost attachments are quite large i cannot even find the posts where it was inserted...
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @3rd Angle I would have to thanks @MattW for helping you with the issue. Thank you @MattW!

      Do you happen to know why the attachment files are missing? They should not be missing at all! And skipping files are quite dangerous because it may cause trouble later on. I'm not sure what is the best way to deal with your situation... You said you lost many attachments? How many is that? I'm hoping a number less than 50...
       
  8. 3rd Angle

    3rd Angle New Member

    I had migrated server recently and had deleted few attachments which were not a part of this domain name i am using this addon for. So i really dont know the exact issue..

    I can pass on the login credentials in private if you could look into it for me..
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @3rd Angle Yes, I would love to take a look. Thank you.
       
  9. 3rd Angle

    3rd Angle New Member

    Thanks for the support.... it is done now...

    In case others are facing the same issue, reducing the Items to process to 5 solved it for me....
     
    xfrocks likes this.
  10. David B. Bitton

    David B. Bitton New Member

    I was looking in my attachments folder on the local server, and I found 2 png files in data/attachments/26. Is there a tool that will "fix" this and move them to S3? Also, is there a solution for avatars? I really want to be able to serve all dynamic content via S3/CloudFront.
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @David B. Bitton You may run the move tool in AdminCP > Tools > Rebuild Caches. It will move old attachments to S3.
       
  11. Price06

    Price06 New Member

    #48 Price06, Oct 29, 2015
    Last edited: Oct 29, 2015
    Hi,

    I bought and spent nearly half a day to activate this plugin. At the end of the day, I am always getting this error (s3 + CloudFront and I don't have any other addon):

    XenForo_Exception: Failed to write the attachment file. - library/XenForo/DataWriter/AttachmentData.php:146
    #0 /usr/share/nginx/www/forum/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(55): XenForo_DataWriter_AttachmentData->_postSave()
    #1 /usr/share/nginx/www/forum/library/XenForo/DataWriter.php(1409): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave()
    #2 /usr/share/nginx/www/forum/library/XenForo/Model/Attachment.php(539): XenForo_DataWriter->save()
    #3 /usr/share/nginx/www/forum/library/XenForo/ControllerPublic/Attachment.php(204): XenForo_Model_Attachment->insertUploadedAttachmentData(Object(XenForo_Upload), 1)
    #4 /usr/share/nginx/www/forum/library/XenForo/FrontController.php(347): XenForo_ControllerPublic_Attachment->actionDoUpload()
    #5 /usr/share/nginx/www/forum/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
    #6 /usr/share/nginx/www/forum/index.php(13): XenForo_FrontController->run()
    #7 {main}
    array(3) {
    ["url"] => string(110) "http://DOMAINNAME.com/forum/attachm...81a24176881f190aed8101dfd27&content_type=post"
    ["_GET"] => array(3) {
    ["/forum/attachments/do-upload_json"] => string(0) ""
    ["hash"] => string(32) "6cf9f81a24176881f190aed8101dfd27"
    ["content_type"] => string(4) "post"
    }
    ["_POST"] => array(7) {
    ["Filename"] => string(13) "kiyaslama.jpg"
    ["_xfSessionId"] => string(32) "cc9a863d26614b58fa722781f35122dc"
    ["_xfToken"] => string(8) "********"
    ["_xfResponseType"] => string(4) "json"
    ["content_data"] => array(1) {
    ["node_id"] => string(2) "47"
    }
    ["_xfNoRedirect"] => string(1) "1"
    ["Upload"] => string(12) "Submit Query"
    }
    }
     
    1. Price06

      Price06 New Member

      Price06 @Price06 Well, I solved the problem myself :D The problem is that this plugin is little bit old. I was trying to use Frankfurt but it is not in region list of this addon. As a result, baaaaam you cannot use it. Select Ireland if you need to use Europe.

      And I see that if you have trouble with this addon, you are alone to solve it. Thank you @xfrocks for no help but sold!
       
      xfrocks likes this.
      1
    2. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Price06 Sorry for your trouble, newer version of the add-on has included Frankfurt but it hasn't been released yet :)
       
  12. Legend's Gate Media

    Legend's Gate Media New Member

    I got the attachments to work great but how can I get the Avatars to work with this as well?
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Legend's Gate Media For avatar (and other files in /data), you need the [bd] Data Storage add-on.
       
  13. Legend's Gate Media

    Legend's Gate Media New Member

    Thank you!
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Legend's Gate Media The guide is a bit outdated, have you had any trouble following it?
       
    2. Legend's Gate Media

      Legend's Gate Media New Member

      Legend's Gate Media @xfrocks It looks like I got it working. Thanks.

      I have both the attachments and data storage add-on running. Do I need to have both or does the data storage add-on also work with the attachments that people upload?
       
    3. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Legend's Gate Media If you want everything on the remote storage then you need both of them. The [bd] Data Storage only handles attachment thumbnail (not the full file) FYI.
       
  14. thenet

    thenet New Member

    Hmm.. I could use a dig out here.

    To rule out some security elements I made the Bucket public.
    https://attachments.useecloud.com/800px-Polaroid_Sunglasses_logo.png

    Couldn't upload .mp4 files to the front end with this plugin enabled.
    Uploading images seems ok.

    e.g. 800px-Polaroid_Sunglasses_logo.png - OK
    Fossetts_2018.mp4 =
    The following error occurred
    There was a problem uploading your file.

    Fossetts 2018.mp4
     
  15. thenet

    thenet New Member

    Ok managed to whittle it down and create a new Access Key etc.
    How can I make a video that gets uploaded reside in the S3 folder also?
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @thenet So your files are uploaded okie now?
      What did you mean when you say "reside in the S3 folder"? If they are uploaded to S3, they should be in the S3 bucket already?
       
Loading...