[bd] Attachment Store [Paid] 1.3.0

Store attachment differently and more effectively.

  1. xfrocks

    xfrocks XenForo rocks! Staff Member

    That would be a nice approach for this problem but we generally want to avoid making joined query or extra one :(
     
  2. joec

    joec New Member

    Why? An extra JOIN on an indexed column shouldn't cause any appreciable performance decrease, especially considering that the data in xf_bd_attachment_store will almost always be cached already. It should actually be faster than duplicating the same data in every row of xf_attachment_data.
     
  3. xfrocks

    xfrocks XenForo rocks! Staff Member

    Most likely we will need an extra query because the attachment fetch query doesn't support join properly, some hack will need to be done to do that. Also, we prefer to have configuration like this as environment variable so config.php based option is better in that regard. If you really like to have it in a separate table, we may offer both ways actually.
     
  4. joec

    joec New Member

    True, I had to modify library/XenForo/Model/Attachment.php to support the extra JOIN clause, but on the other hand it's a very straightforward change (and not too hackish, since it's very similar to how $dataColumns works). How would I propose a change like that to the XF devs?
    Could you give more details on this approach?
     
  5. xfrocks

    xfrocks XenForo rocks! Staff Member

    Probably the suggestion forum. The use case for this is quite limited though so I'm not sure whether they would consider it.

    Basically the idea is to version control the entire site which means putting all .php and assets etc in a repository. However, keeping credentials (db username password, S3 keys etc.) in that one repo is not a good idea... A way to do it is to define those information as environment variable and "import" them in config.php (using getenv() or similar functions).
     
  6. joec

    joec New Member

    I made the suggestion, but it wasn't accepted. However I was able to implement the JOIN clause without modifying core XF files, by overriding some methods in Attachment.php. There's some duplicated code, but not that much. Again, just let me know if you're interested in seeing my modifications.

    So bdAttachmentStore would use the same set of credentials for every image? That would be an improvement, but not a complete solution if the other config options like bucket name and region still get duplicated in every xf_attachment_data row.

    BTW, have you looked at using instance profile credentials for S3? It's the preferred way to do permissions nowadays. The AWS SDK for PHP has built-in support for that, though I don't know whether XF addons can use external dependencies.
     
  7. joec

    joec New Member

    A separate issue I just noticed is that the way this addon names objects in S3 buckets could cause a performance problem. Currently it uses the key naming scheme YYYY/MM/NN_XXXX, where YYYY/MM is the year and month of the upload, NN is the attachment's data_id and XXXX is the MD5 hash of the attachment data. But sequential key prefixes are bad for performance on S3. Since the YYYY/MM prefix is not really necessary, as S3 isn't a hierarchical file system and has no maximum number of files per "directory" (since there really are no directories on S3), I would recommend just using the "flat" naming scheme XXXX_NN (hash followed by data_id), to ensure attachments are uniformly distributed among S3's partitions.
     
  8. xfrocks

    xfrocks XenForo rocks! Staff Member

  9. xfrocks

    xfrocks XenForo rocks! Staff Member

    v1.3.0 is out with a new feature which will let you put the credentials in config.php and keep them out of db. Make sure you try that.
    Regarding instance profile credentials, you meant IAM? It's possible to use IAM key pair with this add-on, in fact it is the recommended way to configure S3 access in our guides (here and here).

    Unfortunately the directory structure is chosen so it can be used with both local file system (external data, ftp mode) and remote (S3) allowing easy migration of large amount of data. We may support flat file structure in later release though.
     
    Fred. likes this.
  10. osxinfo

    osxinfo New Member

    Hello there
    I get the following error after installing the plugin.

    Code:
    XenForo_Exception: Failed to write the attachment file. - library/XenForo/DataWriter/AttachmentData.php:146
    
    Stack Trace
    #0 /home/osxinfon/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(55): XenForo_DataWriter_AttachmentData->_postSave()
    #1 /home/osxinfon/public_html/library/XenForo/DataWriter.php(1423): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave()
    #2 /home/osxinfon/public_html/library/XenForo/Model/Attachment.php(539): XenForo_DataWriter->save()
    #3 /home/osxinfon/public_html/library/bdAttachmentStore/XenForo/Model/Attachment.php(68): XenForo_Model_Attachment->insertUploadedAttachmentData(Object(XenForo_Upload), 1, Array)
    #4 /home/osxinfon/public_html/library/XenForo/ControllerPublic/Attachment.php(205): bdAttachmentStore_XenForo_Model_Attachment->insertUploadedAttachmentData(Object(XenForo_Upload), 1)
    #5 /home/osxinfon/public_html/library/XenForo/FrontController.php(351): XenForo_ControllerPublic_Attachment->actionDoUpload()
    #6 /home/osxinfon/public_html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
    #7 /home/osxinfon/public_html/index.php(13): XenForo_FrontController->run()
    #8 {main}
    Request State
    array(3) {
      ["url"] => string(101) "https://osxinfo.net/eklentiler/do-upload.json?hash=d6ce873222a9d9f145154258b87d1aec&content_type=post"
      ["_GET"] => array(2) {
        ["hash"] => string(32) "d6ce873222a9d9f145154258b87d1aec"
        ["content_type"] => string(4) "post"
      }
      ["_POST"] => array(4) {
        ["_xfToken"] => string(8) "********"
        ["_xfNoRedirect"] => string(1) "1"
        ["_xfSessionId"] => string(32) "409687bf6cf97fc299eda3f5f1fe05b2"
        ["content_data"] => array(1) {
          ["thread_id"] => string(3) "344"
        }
      }
    }
    
     
  11. xfrocks

    xfrocks XenForo rocks! Staff Member

    Hi there,

    Which mode are you trying to configure? S3, ftp or external data?
     
  12. osxinfo

    osxinfo New Member

    Hi, Amazon S3 methot.
     
  13. osxinfo

    osxinfo New Member

  14. osxinfo

    osxinfo New Member

  15. osxinfo

    osxinfo New Member

    I bought this plug-in but I can not use it. Because the developer has not yet responded to the problem that I have shared in the previous message. Nice support!
     
  16. xfrocks

    xfrocks XenForo rocks! Staff Member

    Sorry for the late response. It's likely an issue with how you configured the S3 key/secret. Have you been able to verify your key pair separately (via command line or other tool)? Also, enabling XenForo debug mode can reveal some extra information in ./internal_data/bdAttachmentStore_ShippableHelper_S3.log file.
     
  17. osxinfo

    osxinfo New Member

    @xfrocks
    I've changed AWS keys and The delay process is active. Files are added correctly. However, when moves files to S3 all images disappear.
    Screen Shot 2017-02-20 at 11.42.40.png
     
    Last edited: Feb 20, 2017
  18. joec

    joec New Member

    It's a good feature, though not quite what we need. But I'm currently planning on running a customized version of this addon anyway, so no worries.

    No, I mean instance profile credentials, which allow an EC2 instance to assume an IAM role without ever needing the access key id and secret key. It's only applicable to EC2 instances of course, and it's more of a "would be nice" feature than any kind of requirement.

    Understood. BTW, I noticed another issue with key naming: thumbnails are always given the extension .jpg, even though XF generates thumbnails in the same format as the original attachment. This means we can end up with PNG or GIF thumbnails using the .jpg extension, which causes S3 to serve them with the wrong MIME type. Browsers seem to handle this fine, but it should probably be fixed at some point.
     
  19. osxinfo

    osxinfo New Member

    @xfrocks What do we need to do to get a quick answer? :)
     
  20. hoangnamdoan33

    hoangnamdoan33 New Member

    anh em sử dụng amz c3 giờ muốn chuyển về localhosst thì phải làm sao ạ
     
Loading...