[bd] Attachment Store for XenForo 1 [Paid] 1.3.7

Store attachment differently and more effectively.

  1. Nano2nd

    Nano2nd New Member

    so my latest issue is that the move process keeps failing, i've got 117G worth of attachments and its taken over an hour to get to 180MB! can this be run at the command line? any idea why its failing? there nothing in the logs...
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Nano2nd The move to external is failing? That shouldn't happen too many times because local file move is basically instantaneous.
       
    2. Nano2nd

      Nano2nd New Member

      Nano2nd @xfrocks yes that right, it flys upto about 30,000 then just crawls and eventually fails.... restarting it a few times just seems to result in failure as well :(
       
    3. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Nano2nd It "flies" because it knows which files have been moved so they are skipped. How many do you have? Also, no server error logs anything? :eek:
       
  2. Nano2nd

    Nano2nd New Member

    hi, yeah i've converted 1.4m images ( forum is 16 years old) myself, thats not the issue, its the new stuff that didn't seem to work i find that convertimage doesn't catch everything if a user uploads lots of images, so i use convertimageall to mop up the ones that get missed... i might stop using convertimage and just to a single run of convertimageall once a day.

    my issue at the moment is this attachment addon doesn't seem to work, it fails and theres no errors, i'm hoping theres some way to debug it
     
  3. cjwinternet

    cjwinternet New Member

    Actually, when I said bd worked perfectly, the end result was perfect, but it did stall more than a few times each day.

    After a day or so i started an Amazon server instance and kept an eye on the process via RDP, minimising downtime.
     
    xfrocks likes this.
  4. Nano2nd

    Nano2nd New Member

    Yeah nothing in the logs at all, there's over a million images
     
  5. hoangnamdoan33

    hoangnamdoan33 New Member

    có thể bảo mật liên kết tải về (ftp) không ad :(
     
  6. Yodrak

    Yodrak New Member

    @xfrocks I bought your Attachment Store Addon but having problems to export all internal_data Files in the data Directory in my Testenvironment. after around 30000 Attachments moved it crash and if i want start the rebuild again then the Rebuild Button for Attachment Store is missing.
     
  7. TheComputerGuy

    TheComputerGuy New Member

    Hi there - I'm running 1.3.0. I need to uninstall it, as my AWS bill was too high. I'm hosting both files on locally, and pushed to S3

    How do I uninstall?

    These are the steps I've taken so far:
    Changed the Default.
    Ran tool to update, and nothing seems to update the attachment location.
     
  8. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    RoldanLT likes this.
    1. Nano2nd

      Nano2nd New Member

    2. Nano2nd

      Nano2nd New Member

      Nano2nd @Nano2nd any help on this?
       
    3. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Nano2nd Please try re-uploading the files. Some are missing I think.
       
  9. RoldanLT

    RoldanLT Active Member

    Is this enabled by default?
    Where can I see this?
     
  10. alfa1

    alfa1 Active Member

    There is a conflict between bs attachment store and Lazy Loader:
    https://xenforo.com/community/resources/lazy-load-img.3979/

    Unmaintained - Lazy Load [img]

    Provides lazy loaded image support via the Lazysizes A zero query method for per user-group lazy loading of the [img] and [plain] Enable Outside threads/Conversations permits the lazy loading...
    xenforo.com


    Thumbs fail for all new images added since I ran the combination of lazy load, attachment improvements and bd attach.
    When I disabled lazy load and attachment improvements and rebuild thumbs then everything started working again.

    Additionally this causes some flickering of IPs which causes IPs to get banned by our firewall.
     
  11. pete

    pete New Member

    yeah something's not right. i'm using a different lazy load but disabling it makes no difference - with delayed processing enabled, attachments upload and work briefly before vanishing from the post.


    With processing not delayed they can't be uploaded at all and fail with

    Code:
    The following error occurred
    There was a problem uploading your file. 

    Rebuilding thumbnails results in many many many of these:
    Code:
    Error Info
    XenForo_Exception: Thumb rebuild for #7553: Failed to write the attachment thumbnail data. - library/XenForo/DataWriter/AttachmentData.php:169
    Generated By: admin, A moment ago
    Stack Trace
    #0 /home/thumped/public_html/bbs/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(54): XenForo_DataWriter_AttachmentData->_postSave()
    #1 /home/thumped/public_html/bbs/library/XenForo/DataWriter.php(1423): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave()
    #2 /home/thumped/public_html/bbs/library/XenForo/Deferred/AttachmentThumb.php(67): XenForo_DataWriter->save()
    #3 /home/thumped/public_html/bbs/library/XenForo/Model/Deferred.php(295): XenForo_Deferred_AttachmentThumb->execute(Array, Array, 7.9999990463257, '')
    #4 /home/thumped/public_html/bbs/library/XenForo/Model/Deferred.php(429): XenForo_Model_Deferred->runDeferred(Array, 7.9999990463257, '', false)
    #5 /home/thumped/public_html/bbs/library/XenForo/Model/Deferred.php(374): XenForo_Model_Deferred->_runInternal(Array, 8, '', false)
    #6 /home/thumped/public_html/bbs/library/XenForo/ControllerAdmin/Tools.php(159): XenForo_Model_Deferred->run(true, NULL, '', false)
    #7 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(351): XenForo_ControllerAdmin_Tools->actionRunDeferred()
    #8 /home/thumped/public_html/bbs/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
    #9 /home/thumped/public_html/bbs/admin.php(13): XenForo_FrontController->run()
    #10 {main}
    Request State
     
  12. Triops

    Triops New Member

    #503 Triops, Dec 2, 2017
    Last edited: Dec 2, 2017
    Post removed, @xfrocks answered on xenforo.com. Thank you!
     
  13. Dryline

    Dryline New Member

    Currently running version 1.3.5b, but discovered after updating to version 1.3.4 on December 1, 2017, attachments stopped being moved to the S3 bucket I have set up for them. The last attachment in the S3 bucket is dated 11/30/17, so one could logically assume something went haywire with the 1.3.4 update.

    Is there anything in the 1.3.4 code update that you think may have caused the attachments to stop being transferred to S3?
     
  14. Dryline

    Dryline New Member

    This is from the S3 Shippable Helper log. It’s shows the last attachment that moved successfully to S3 on 11/13/17 to the first one that didn’t on 12/7/17. It looks like the “Host” changed between version 1.3.0 to 1.3.4. I hope this helps, so I can get the advice necessary to fix it, whether it’s on my end or yours.

    2017-11-13 15:11:21 DELETE http://s3-us-west-2.amazonaws.com:80/st-attach/2017/11/18508_9656a0bfdd5ed58110b951a054241769_thumb.jpgarray (
    'x-amz-content-sha256' => 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855',
    'x-amz-date' => '20171113T151121Z',
    'Host' => 's3-us-west-2.amazonaws.com',
    'Authorization' => 'AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXX/20171113/us-west-2/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=38a029234ed0f362d093417f1b102abb3db9467a4cf7ac8f3b8044a4a0f52a49',
    ) -> 204


    2017-12-07 19:56:08 PUT http://s3.amazonaws.com:80/st-attach/2017/12/18517_0e84b8c461c62b4404d72b86ebb78314.jpeg array (
    'Content-Type' => 'image/jpeg',
    'x-amz-acl' => 'public-read',
    'Expect' => '100-continue',
    'x-amz-content-sha256' => '6bbf5f1c617931ab7bde51738fb49f0928eeefa4ab87b8f1494ee117a5b17474',
    'x-amz-date' => '20171207T195607Z',
    'Host' => 's3.amazonaws.com',
    'Authorization' => 'AWS4-HMAC-SHA256 Credential=XXXXXXXXXXXXXXX/20171207/us-east-1/s3/aws4_request,SignedHeaders=content-type;expect;host;x-amz-acl;x-amz-content-sha256;x-amz-date,Signature=8db89a21dc8024162f5fe0ca6e286658765a08015b9a240350f0e9a4e4d14e1d',
    'Content-type' => 'image/jpeg',
    ) -> 301 <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>st-attach</Bucket><Endpoint>st-attach.s3.amazonaws.com</Endpoint><RequestId>07615445EFA1CFE7</RequestId><HostId>DSgCpuWDm//LAm2ZEUuJxRxmlgFMfEor6lHNWkp9zaACApPN538dagRsVliG4UVlgaeHAqOnXAQ=</HostId></Error>
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Dryline Are you using the delayed processing feature? Also, can you double check the region option in AdminCP? The old log entry indicates you were using us-west-2, the new one implies it was using us-east-1...
       
  15. fmsc88

    fmsc88 New Member

    Hi there,

    I’ve been trying out the add-on and so far it’s working pretty fine with Amazon S3. One thing I would like to know is if there any way to preserve the filenames of the files uploaded to S3, so instead of having a filename composed by ID_HASH.zip to have the OriginalName.zip. Is there any option or configuration of the module that allows this or can it be done via code?

    Best
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @fmsc88 There is an option in AdminCP > Options > Attachments to maintain filename. I think that would work for you.
       
    2. fmsc88

      fmsc88 New Member

      fmsc88 @xfrocks Hello,basically that checkbox is missing, missing file name.JPG

      I can see that checkbox when i choose "Store file in remote server (via FTP)" filename.JPG
       
  16. fmsc88

    fmsc88 New Member

  17. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    Ah, sorry. For S3, the filename is put in the metadata so when you download it, the actual filename gets saved on the computer.
     
    1. fmsc88

      fmsc88 New Member

      fmsc88 @xfrocks Yeah when i download on the pc the file has the correct filename,isn't there any way to have the correct filename and on S3?
      In my opinion is useless if all uploads will be like this files.JPG because how i'll delete old versions on S3?
       
  18. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    You should never delete files from storage manually, the add-on will delete files for deleted attachments by itself. In worst case scenarios, you can depend on the data_id to delete files. But again, the add-on will do that for you.
     
    1. fmsc88

      fmsc88 New Member

      fmsc88 @xfrocks @xfrocks all i want to know is how can i keep the file-name.zip at s3 instead of hash,it is this possible or not?
      That's the reason i bought the add-on yesterday,come on...
       
    2. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @fmsc88 If you insist, I can make the change to keep the filename in S3. However, doing so actually make it harder to manage the files. And your reason to need filenames on S3 is not a strong one (e.g. files must not be deleted manually, the system will do it anyway).
       
    3. fmsc88

      fmsc88 New Member

      fmsc88 @xfrocks For how long the attachment should be unused so the con to delete the attachment on s3?
       
    4. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @fmsc88 If it is unused at the time the cron runs, it will be deleted right away. No wait time.
       
Loading...