[HOWTO] XenForo attachments via Amazon S3 and CloudFront

Discussion in 'Other' started by xfrocks, May 3, 2013.

  1. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    Prerequisite:
    Step 1. Obtain Amazon Web Services Security Credentials + CloudFront Key Pair
    Follow step 1 here: https://xfrocks.com/other/threads/howto-xenforo-attachments-via-amazon-s3-and-cloudflare.59/

    [HOWTO] XenForo attachments via Amazon S3 and CloudFlare

    Prerequisite: XenForo 1.1.4+ [bd] Attachment Store 0.9.5+ Step 1. Obtain Amazon Web Services Security Credentials Go to...
    xfrocks.com


    Also at https://portal.aws.amazon.com/gp/aws/securityCredentials, choose the tab "Key Pairs", you will be able to create a new key pair. You will be asked to save the private key file and given a Key Pair ID. Save the .pem file to a safe directory, we will use it later.

    Step 2. Create an Amazon S3 bucket
    Follow the step 2 here: https://xfrocks.com/other/threads/howto-xenforo-attachments-via-amazon-s3-and-cloudflare.59/

    [HOWTO] XenForo attachments via Amazon S3 and CloudFlare

    Prerequisite: XenForo 1.1.4+ [bd] Attachment Store 0.9.5+ Step 1. Obtain Amazon Web Services Security Credentials Go to...
    xfrocks.com


    Step 3. Create an Amazon CloudFront Distribution
    Go to https://console.aws.amazon.com/cloudfront/home and click the "Create Distribution" button. You will need to fill a rather long form with lots of fields but some important things to remember:
    • Choose "Download" as the delivery method
    • Enter the full Amazon S3 domain name as the "Origin Domain Name". As you type, auto suggestion will showup, just choose from the list
    • Choose "Yes" for "Restrict Bucket Access" and "Restrict Viewer Access (Use Signed URLs)". See the below screenshot to configure your distribution:
    Screen Shot 2013-05-04 at 2.01.23 AM.png
    Screen Shot 2013-05-04 at 2.03.32 AM.png

    Step 4. Configure [bd] Attachment Store

    Follow step 3 here: https://xfrocks.com/other/threads/howto-xenforo-attachments-via-amazon-s3-and-cloudflare.59/

    [HOWTO] XenForo attachments via Amazon S3 and CloudFlare

    Prerequisite: XenForo 1.1.4+ [bd] Attachment Store 0.9.5+ Step 1. Obtain Amazon Web Services Security Credentials Go to...
    xfrocks.com


    Enable the option "Use Amazon CloudFront to serve attachments" and fill in information from CloudFront console. You will need to open the .pem file with a text editor (Notepad or TextEdit) and copy/paste its contents into the box.

    Screen Shot 2013-05-04 at 2.06.34 AM.png

    Done :D
     
  2. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    Hey Andy, have you been able to setup Amazon S3 and CloudFront?
     
    1. Andy

      Andy Member

      Andy @xfrocks Not yet. Plan to play with it this weekend. I have to disable Dinh Thanh's addon first.
       
    2. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Andy I have heard that they can run together without issues (so far).
       
  3. cmeinck

    cmeinck New Member

    Where do I find the .pem file and the additional settings for key pair id and private key?

    I've check "Keep Local Copy". Is it correct that I should uncheck "Set Private Access"? I'd like to be able to revert to local storage if this doesn't work out well.

    Using XenMedia Gallery, I'm seeing the following errors:
    openssl_sign() [function.openssl-sign]: supplied key param cannot be coerced into a private key
    1. XenForo_Application::handlePhpError()
    2. openssl_sign() in bdAttachmentStore/Helper/CloudFront.php at line 43
    3. bdAttachmentStore_Helper_CloudFront::_sign() in bdAttachmentStore/Helper/CloudFront.php at line 21
    4. bdAttachmentStore_Helper_CloudFront::getSignedUrl() in bdAttachmentStore/Model/File.php at line 178
    5. bdAttachmentStore_Model_File->_s3_getFileUrl() in bdAttachmentStore/Model/File.php at line 59
    6. bdAttachmentStore_Model_File->getFileUrl() in bdAttachmentStore/XenForo/Model/Attachment.php at line 58
    7. bdAttachmentStore_XenForo_Model_Attachment->getAttachmentThumbnailUrl() in XenForo/Model/Attachment.php at line 372
    8. XenForo_Model_Attachment->prepareAttachment() in XenForo/Model/Attachment.php at line 403
    9. XenForo_Model_Attachment->prepareAttachments() in XenGallery/ControllerPublic/Media.php at line 49
    10. XenGallery_ControllerPublic_Media->actionIndex() in XenForo/FrontController.php at line 313
    11. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 132
    12. XenForo_FrontController->run() in /home/everythc/public_html/forum/index.php at line 13
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @cmeinck You can get the key info in step 1 of this howto.

      You can both have the file locally + set private. They are unrelated options.

      You had the exception because of misconfiguration, you haven't copied the pem file contents right?
       
  4. cmeinck

    cmeinck New Member

    I was able to get the key pairs for Cloudfront and the private key from the .pem. I'm not seeing the same errors, but I cannot upload any images to the site or to the gallery.

    What does it mean when you say, "..make it impossible to turn off CloudFront later." Can you clarify?

    Hard to share my info, but here's my setup.

    settings.png
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @cmeinck What is the new error message?

      About impossible to disable CloudFront, that means: once you enable the set private option, your file in S3 will be set to private and only accessible from CF servers. If you disable CF later, those files cannot be served directly to users. There are ways to make them accessible but doing that is rather complicated.
       
  5. cmeinck

    cmeinck New Member

    Just to clarify, no changes are needed to my host DNS or Cloudfare (which I don't use).
     
  6. cmeinck

    cmeinck New Member

    It just won't allow me to upload an image. My server log shows the following:

    ErrorException: openssl_sign() [<a href='function.openssl-sign'>function.openssl-sign</a>]: supplied key param cannot be coerced into a private key - library/bdAttachmentStore/Helper/CloudFront.php:43

    So that's probably not a good option to check. I would always like the option to turn it off and still have my media accessible locally.


     
  7. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    That means you copied the .pem contents incorrectly (included line break or space etc.)

    The "set private" option only affects your files in S3. If you have "keep file locally" option enabled, you will still be able to turn if off anytime.
     
    1. cmeinck

      cmeinck New Member

      cmeinck @xfrocks Tried it three times again. No luck. I'm using Coda (Mac editor) and I'm not seeing any line breaks or space when pasting into the options panel.

      Same error in the server logs.
       
    2. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @cmeinck If you want, you can send me the .pem file via a conversation (you will need to zip it first) and your site Administrative account. I will come and see if there is any problem. You can always delete the key and generate a new one later.
       
  8. TheBigK

    TheBigK New Member

    @xfrocks - I followed all the steps above, but the attachments aren't being served from data.crazyengineers.com. I've setup CloudFront before; so I'm sure I haven't missed any step.

    I tried rebuilding attachments - but got following error: An error occurred or the request was stopped.
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @TheBigK Have you tried uploading a new file to see if it works?

      Also, please check server log for error messages.
       
  9. TheBigK

    TheBigK New Member

    Before I go ahead with this - Does this addon have the functionality offered by -http://xenforo.com/community/threads/tinhte-image-attachment-optimization.34400/ ?

    I only tried with the existing image attachements. Do I need to run attachment rebuild first? It's extremely slow!
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @TheBigK Not exactly the functionality of that add-on but this offers caching in a different way.

      It runs slow because it has to upload to S3. Depending on the attachment size and number of files it will take a long long time.
       
  10. Fred.

    Fred. New Member

    I did all of this.
    I'm using S3 + cloudfront from a static cookieless domain.

    Upload is not working and i get this error.
    I'm trying to figure it out for a long time now, but I don't know what's wrong. :confused:

    Code:
    Server Error Log
    Error Info
    XenForo_Exception: Failed to write the attachment file. - library/XenForo/DataWriter/AttachmentData.php:145
    Generated By: Fred., Yesterday at 7:27 PM
    Stack Trace
    #0 /home/admin/domains/domain.com/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(63): XenForo_DataWriter_AttachmentData->_postSave()
    #1 /home/admin/domains/domain.com/public_html/library/XenForo/DataWriter.php(1397): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave()
    #2 /home/admin/domains/domain.com/public_html/library/XenForo/Model/Attachment.php(524): XenForo_DataWriter->save()
    #3 /home/admin/domains/domain.com/public_html/library/XenForo/ControllerPublic/Attachment.php(201): XenForo_Model_Attachment->insertUploadedAttachmentData(Object(XenForo_Upload), 3)
    #4 /home/admin/domains/domain.com/public_html/library/XenForo/FrontController.php(347): XenForo_ControllerPublic_Attachment->actionDoUpload()
    #5 /home/admin/domains/domain.com/public_html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
    #6 /home/admin/domains/domain.com/public_html/index.php(13): XenForo_FrontController->run()
    #7 {main}
    Request State
    array(3) {
      ["url"] => string(108) "http://www.domain.com/attachments/do-upload.json?hash=7c58551f769b4ecd7017ed2c6ff3e75a&content_type=post"
      ["_GET"] => array(2) {
        ["hash"] => string(32) "7c58551f769b4ecd7017ed2c6ff3e75a"
        ["content_type"] => string(4) "post"
      }
      ["_POST"] => array(7) {
        ["Filename"] => string(18) "logo.png"
        ["_xfToken"] => string(8) "********"
        ["content_data"] => array(1) {
          ["thread_id"] => string(5) "10355"
        }
        ["_xfResponseType"] => string(4) "json"
        ["_xfNoRedirect"] => string(1) "1"
        ["_xfSessionId"] => string(32) "331a26ad98b077235a9bd4d45fde5fac"
        ["Upload"] => string(12) "Submit Query"
      }
    }
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Fred. Do you enable local copy option?
       
  11. Fred.

    Fred. New Member

    Yes, I think i got it working now. I just uploaded an image to s3. But it looks like it has another folder structure? Do i need to copy anything to S3? Because only the uploaded file is there.
     
  12. Fred.

    Fred. New Member

    I think i got it... rebuilding caches now
     
  13. Fred.

    Fred. New Member

    Ok, It works now.

    But i still shows the cloudfront url. I want to use my custom url att.staticdomain.com

    I've made the CNAME record (a while ago) and added the domain as alternate domain in cloudfront, but if i change the cloudfront domain to att.staticdomain.com in XenForo it doesn't work anymore. :confused:
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Fred. CNAME records take time to work itself across the globe.
       
    2. Fred.

      Fred. New Member

      Fred. @xfrocks The CNAME is updated. It shows the correct A records to cloudfront. There must be something else wrong.
      If i try to link to an image with the cloudfront url in the browser everything works perfect. But with the custom url it's not working.
      I've done this before without any problems. :confused:

      Now it shows this with the custom url if i try it in the browser.

      Code:
      This XML file does not appear to have any style information associated with it. The document tree is shown below.
      <Error>
      <Code>AccessDenied</Code>
      <Message>Access denied</Message>
      </Error>
       
    3. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Fred. Amazon requires the record to be configured correctly otherwise it will not work. You may want to start a conversation with me with detailed information, I will take a look.
       
  14. Fred.

    Fred. New Member

    Thanks xfrocks but I have to go now. I will have another look later. If i'm not able to solve it i'll contact you.

    Now it shows
    Code:
    404 Not Found
    
    Code: NoSuchWebsiteConfiguration
    Message: The specified bucket does not have a website configuration
    BucketName: att.staticdomain.com
    RequestId: D3B8E2AC3F0B9CA4
    HostId: fpm6NHliq8aFFkA2mCj9o5zznIo+qm2JQkqT0PVm0kDARP0O72xRlzwzkZo/B2d6
    :confused:
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Fred. Most likely a misconfiguration between CloudFront and S3. Have you tried using S3 only? We should test step by step ;)
       
  15. Russoroni

    Russoroni New Member

    Hello again @xfrocks ok so regarding cloudfront

    What does this error mean?

    Server Error
    openssl_sign(): supplied key param cannot be coerced into a private key

    1. XenForo_Application::handlePhpError()
    2. openssl_sign() in bdAttachmentStore/Helper/CloudFront.php at line 43
    3. bdAttachmentStore_Helper_CloudFront::_sign() in bdAttachmentStore/Helper/CloudFront.php at line 21
    4. bdAttachmentStore_Helper_CloudFront::getSignedUrl() in bdAttachmentStore/Model/File.php at line 195
    5. bdAttachmentStore_Model_File->_s3_getFileUrl() in bdAttachmentStore/Model/File.php at line 79
    6. bdAttachmentStore_Model_File->getFileUrl() in sonnb/XenGallery/Model/ContentData.php at line 436
    7. sonnb_XenGallery_Model_ContentData->getContentDataLargeThumbnailUrl() in sonnb/XenGallery/Model/Content.php at line 365
    8. sonnb_XenGallery_Model_Content->prepareContent() in sonnb/XenGallery/Model/Content.php at line 439
    9. sonnb_XenGallery_Model_Content->prepareContents() in sonnb/XenGallery/Model/Album.php at line 356
    10. sonnb_XenGallery_Model_Album->attachCoversToAlbums() in sonnb/XenGallery/ControllerPublic/XenGallery.php at line 57
    11. sonnb_XenGallery_ControllerPublic_XenGallery->actionIndex() in XenForo/FrontController.php at line 347
    12. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
    13. XenForo_FrontController->run() in /home/trailpeople/public_html/index.php at line 13
     
  16. Russoroni

    Russoroni New Member

    And this isnt a xengallery thing. It happens if I go into any forums too.

    Server Error
    openssl_sign(): supplied key param cannot be coerced into a private key

    1. XenForo_Application::handlePhpError()
    2. openssl_sign() in bdAttachmentStore/Helper/CloudFront.php at line 43
    3. bdAttachmentStore_Helper_CloudFront::_sign() in bdAttachmentStore/Helper/CloudFront.php at line 21
    4. bdAttachmentStore_Helper_CloudFront::getSignedUrl() in bdAttachmentStore/Model/File.php at line 195
    5. bdAttachmentStore_Model_File->_s3_getFileUrl() in bdAttachmentStore/Model/File.php at line 79
    6. bdAttachmentStore_Model_File->getFileUrl() in bdAttachmentStore/XenForo/Model/Attachment.php at line 81
    7. bdAttachmentStore_XenForo_Model_Attachment_Base->getAttachmentThumbnailUrl() in XenForo/Model/Attachment.php at line 407
    8. XenForo_Model_Attachment->prepareAttachment() in XenForo/Model/Post.php at line 456
    9. XenForo_Model_Post->getAndMergeAttachmentsIntoPosts() in XenForo/ControllerPublic/Thread.php at line 72
    10. XenForo_ControllerPublic_Thread->actionIndex() in Waindigo/CustomFields/Extend/XenForo/ControllerPublic/Thread.php at line 16
    11. Waindigo_CustomFields_Extend_XenForo_ControllerPublic_Thread->actionIndex() in EWRporta/ControllerPublic/Thread.php at line 9
    12. EWRporta_ControllerPublic_Thread->actionIndex() in sonnbProfanityFilter/ControllerPublic/Thread.php at line 17
    13. sonnbProfanityFilter_ControllerPublic_Thread->actionIndex() in ShowSimilarThreads/ControllerPublic/Thread.php at line 7
    14. ShowSimilarThreads_ControllerPublic_Thread->actionIndex() in XenForo/FrontController.php at line 347
    15. XenForo_FrontController->dispatch() in XenForo/FrontController.php at line 134
    16. XenForo_FrontController->run() in /home/trailpeople/public_html/index.php at line 13
     
  17. Russoroni

    Russoroni New Member

    messed up part about this is that I had no issues at all the first time I setup cloudfront a couple months ago. Just followed your instructions. All I did was repeat and now I get this error. I tried pasting the contents of the key file in as-is and without the <------------START----------> But it's a no-go.
     
  18. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    This is a problem with the .pem file. It should be copied as is (no removal, no additional line break etc.)
     
    Russoroni likes this.
  19. Russoroni

    Russoroni New Member

    Ok so I created a new cloudfront key and pasted that into attachment store and it works. Not sure why the old one didn't work anymore but whatever... So now I create a thread and upload------> all attachment thumbnails are broken----->inserted images/theyre broken links. I checked the delayed processing option and now stuff gets uploaded locally obviously but if I go to the Amazon S3 management console nothing got uploaded.
     
  20. Russoroni

    Russoroni New Member

    Ok so now I just saw bd attachemnet store come along and take the file off to amazon and delete the local. Now I have a broken image. I know S3 works because I've had it working with attachemnt store for a while with no issues.
     
Loading...