[bd] Attachment Store for XenForo 1 [Paid] 1.3.5c

Store attachment differently and more effectively.

  1. Nano2nd

    Nano2nd Webmaster

    bought this today and been playing with it on my test server, whilst the core functionality works okay can someone explain how i manually move my 500,000 images to S3 - i've no issue using the AWS s3 cli - i just need to understand the process... if i copy the internal_data folder as is, which task to i run to relink the images? or is that not how it works?

    also has anyone got any experience using this with ConvertImage & ConvertImageAll? they don't seem to work together... ideally i want everything in S3, both attachments and remote images via the convertimage addons
    kontrabass likes this.
  2. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    The fastest way is to use external data mode, run the move tool so your files are move from internal_data to external one. Afterwards use your favorite tool to sync data/attachment-files to s3 (during that time, just let new attachment save into external data). Running the tool several time to make sure all files are uploaded then switch to s3 mode. Run the storage options tool to point existing attachment to s3 instead of external data. Done.

    It's a bit complicated. An easier way is to switch to s3 mode right away and just run the move tool. The add-on will upload the file one by one. That will be slower than cli tools though.

    Which convert add-on are you asking about? I assume you want to convert IMG bb code urls into attachments? Most tool should work fine with this add-on.
    kontrabass likes this.
  3. kontrabass

    kontrabass New Member

    We use the same convert addons (andyB 's ConvertImage and ConvertImageAll). I don't currently have this addon installed but will want to do the same and move attachments to s3 soon. I'm using ConvertImageAll to convert 2 million hotlinked images to attachments (saving all photobucket images from 2000 to present). I don't see why the addons wouldn't work together, but haven't tried yet.
    xfrocks likes this.
  4. Nano2nd

    Nano2nd Webmaster

    does the external data folder have to be somewhere in particular? just tried that with a /attachments folder and the upload a file process fails

    as above the addons are - andyB 's ConvertImage and ConvertImageAll
  5. Nano2nd

    Nano2nd Webmaster

    actually ignore that i figured it out, so once everything is moved to /data/attachment-files then i copy that to my s3 bucket via the aws s3 cli. once done i run the "Update Attachment Data Storage Options" to me s3 and also change the default attachment mode to s3?
  6. Nano2nd

    Nano2nd Webmaster

    so my latest issue is that the move process keeps failing, i've got 117G worth of attachments and its taken over an hour to get to 180MB! can this be run at the command line? any idea why its failing? there nothing in the logs...
  7. cjwinternet

    cjwinternet Webmaster

    We just imported and moved 1.4m images using attachment store and Convert Image All. Took a week to import the images, then two weeks to upload them.

    bd worked perfectly, convert image all doesn't convert all images, some hosts like imgur don't come over, maybe it'll be updated to do that in future.

    You need to keep 'working the queue' to keep track of where you are, edit and / or ignore posts that won't convert.
  8. Nano2nd

    Nano2nd Webmaster

    hi, yeah i've converted 1.4m images ( forum is 16 years old) myself, thats not the issue, its the new stuff that didn't seem to work i find that convertimage doesn't catch everything if a user uploads lots of images, so i use convertimageall to mop up the ones that get missed... i might stop using convertimage and just to a single run of convertimageall once a day.

    my issue at the moment is this attachment addon doesn't seem to work, it fails and theres no errors, i'm hoping theres some way to debug it
  9. cjwinternet

    cjwinternet Webmaster

    Actually, when I said bd worked perfectly, the end result was perfect, but it did stall more than a few times each day.

    After a day or so i started an Amazon server instance and kept an eye on the process via RDP, minimising downtime.
    xfrocks likes this.
  10. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    The move to external is failing? That shouldn't happen too many times because local file move is basically instantaneous.
  11. Nano2nd

    Nano2nd Webmaster

    yes that right, it flys upto about 30,000 then just crawls and eventually fails.... restarting it a few times just seems to result in failure as well :(
  12. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    It "flies" because it knows which files have been moved so they are skipped. How many do you have? Also, no server error logs anything? :eek:
  13. Nano2nd

    Nano2nd Webmaster

    Yeah nothing in the logs at all, there's over a million images
  14. hoangnamdoan33

    hoangnamdoan33 New Member

    có thể bảo mật liên kết tải về (ftp) không ad :(
  15. Yodrak

    Yodrak Webmaster

    @xfrocks I bought your Attachment Store Addon but having problems to export all internal_data Files in the data Directory in my Testenvironment. after around 30000 Attachments moved it crash and if i want start the rebuild again then the Rebuild Button for Attachment Store is missing.
  16. TheComputerGuy

    TheComputerGuy New Member

    Hi there - I'm running 1.3.0. I need to uninstall it, as my AWS bill was too high. I'm hosting both files on locally, and pushed to S3

    How do I uninstall?

    These are the steps I've taken so far:
    Changed the Default.
    Ran tool to update, and nothing seems to update the attachment location.
  17. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    RoldanLT likes this.
  18. RoldanLT

    RoldanLT Webmaster

    Is this enabled by default?
    Where can I see this?
  19. alfa1

    alfa1 Webmaster

    There is a conflict between bs attachment store and Lazy Loader:

    Lazy Load [img]

    Provides lazy loaded image support via the Lazysizes A zero query method for per user-group lazy loading of the [img] and [plain] Enable Outside threads/Conversations permits the lazy loading...

    Thumbs fail for all new images added since I ran the combination of lazy load, attachment improvements and bd attach.
    When I disabled lazy load and attachment improvements and rebuild thumbs then everything started working again.

    Additionally this causes some flickering of IPs which causes IPs to get banned by our firewall.