[bd] Attachment Store for XenForo 1 [Paid] 1.3.5c

Store attachment differently and more effectively.

  1. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    xfrocks submitted a new resource:

    [bd] Attachment Store (version 0.9.5) - Store attachment differently and more effectively.

    Read more about this resource...
     
  2. xfrocks

    xfrocks XenForo rocks!
    Staff Member

  3. xfrocks

    xfrocks XenForo rocks!
    Staff Member

  4. xfrocks

    xfrocks XenForo rocks!
    Staff Member

  5. hlchia

    hlchia New Member

    Does the keep local file option moves files that are already stored in s3 only back to th local?
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @hlchia Unfortunately not.
       
  6. hlchia

    hlchia New Member

    If I switch to local storage option from S3 and refresh the attachment cache, the attachments will move from S3 to local file storage?
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @hlchia Yes, that's correct.
       
    2. hlchia

      hlchia New Member

      hlchia @xfrocks Thanks, which means in order to use local file option for S3 storage and move attachments that are already stored in S3 only, I will have to do a 2 step process:
      1. Change to local storage only and refresh attachment store so that all current files in S3 will move back to local
      2. Change back to S3 storage and turn-on keep local file option, and refresh attachment store, so that all current and future attachments will be in S3, while a copy will also be available in local
       
    3. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @hlchia Sounds good to me. I will test doing that right now and get back to you. I think it will work.
       
    4. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @hlchia I can confirmed that it works as expected. May take a while depending on the number of attachments you have and your server network.
       
  7. Andy

    Andy Member

    Trying to download the latest version
    This attachment cannot be shown at this time. Please try back later.
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @Andy Sorry for the problem, you should be able to download now. Minor update (v0.9.8.1) has been released to fix a conflict with XenForo Resource.
       
  8. xfrocks

    xfrocks XenForo rocks!
    Staff Member

  9. MattW

    MattW Member

    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @MattW Can you take a screenshot of the option page? If you enter http://724781872.r.cdn77.net as the root URL, it should not append site URL before it.
       
  10. MattW

    MattW Member

    Hmm, I've put the settings back in, rebuilt the caches, and it's all working as expected now :confused:
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @MattW Interesting, you should keep an eye on it ;)
       
      MattW likes this.
      1
  11. MattW

    MattW Member

    Ok, here is something (it also explains what was happening before, as I didn't use http:// on the front of the URL the first time I rebuilt the cache).

    I've just changed the Root URL as I'm now using a CNAME in front of the cdn77 URL. I've saved the option in the ACP, and it's taken:
    View attachment 75

    I've rebuilt the cache, but it's not changing the URL in the database. It's only applying it to new uploaded attachments.

    FTP settings.PNG

    Code:
    "url";s:28:"http://724781872.r.cdn77.net";}
    I can change the settings back to use default, rebuild the cache, and then set back to FTP and re-add all the settings in again, rebuild the cache, and the URL will be correct (and it also re-uploads all the images to the remote FTP server).
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @MattW That explains everything. Yes, the server information is kept for each attachment to make sure existing attachment work when you switch to a second FTP server for example. Or you can have attachments in different storage options at the same time and they will still work.
       
    2. vktechnology

      vktechnology New Member

      vktechnology @xfrocks Hi

      I test with upload pictures and deleted post and wait for 3 hours and all files still in data/attachment-files/2016/12/
       
    3. cjwinternet

      cjwinternet New Member

      cjwinternet @xfrocks If a second ftp server was used, files will be be served from the new and old ftp servers. What about a subsequent cache rebuild? Would it do this one server at a time depending on which one it had the details for at the time?
       
  12. MattW

    MattW Member

    Thanks for reply.

    I'm just trying to figure the best way to try and use this, in conjunction with Showcase and MediaGallery. I've got quite a few attachments:

    Code:
    [~/public_html/internal_data]# du -sh *
    11G    attachments
    I noticed that with this enabled, it's giving the image as the full hashed image file name, rather than the name of the image like the standard XF are presented. Is this where some of the efficiency comes in, because it's not joining the file name?
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @MattW The hash is used as a security measure to make sure the file name is valid to store in different storage system. 11GB is a lot of data, will take quite a long time if you plan to move them FYI :D
       
  13. MattW

    MattW Member

    Yeah, I know :(

    I'm tempted to just use it with External Data directory rather than have it all go via FTP to cdn77

    I've been testing moving between the storage methods, and everything seems to work OK (putting them back in their original locations) once the cache has been re-built.
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @MattW With that much data, I think external data is a good choice. I believe CDN77 will be able to serve from there just fine.
       
  14. MattW

    MattW Member

    Yes, it's pulling the data and serving from the edge locations OK

    Code:
    Request URL:http://dev-cdn.z22se.com/xenforo/data/attachment-files/2013/05/756_cd5cd5fcc216828e5fffeb61c22c8f84.PNG
    Request Method:GET
    Status Code:304 Not Modified
    Request Headersview source
    Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
    Accept-Charset:ISO-8859-1,utf-8;q=0.7,*;q=0.3
    Accept-Encoding:gzip,deflate,sdch
    Accept-Language:en,en-GB;q=0.8
    Cache-Control:max-age=0
    Connection:keep-alive
    Cookie:xf_session=c69fe27d0ae829e9ad5399649a1954d2
    Host:dev-cdn.z22se.com
    If-Modified-Since:Thu, 09 May 2013 09:20:24 GMT
    Referer:http://dev.z22se.com/xenforo/threads/ftp-upload-2.75/
    User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
    Response Headersview source
    Cache-Control:max-age=2592000
    Cache-Control:public, must-revalidate, proxy-revalidate
    Connection:keep-alive
    Date:Thu, 09 May 2013 12:21:39 GMT
    Expires:Sat, 08 Jun 2013 12:18:41 GMT
    Last-Modified:Thu, 09 May 2013 09:20:24 GMT
    Pragma:public
    Server:nginx
    X-Age:12
    X-Cache:HIT
    X-Edge-IP:63.247.91.2
    X-Edge-Location:Atlanta, US
     
    xfrocks likes this.
  15. MattW

    MattW Member

    Migrated over to my live site now ;)

    Code:
    [~/public_html/data/attachment-files]# du -sh *
    785M    2008
    1.6G    2009
    1.5G    2010
    881M    2011
    2.3G    2012
    3.7G    2013
    4.0K    index.html
    One issue I've got is that I adjusted an add-on to pull media from the xenmediagallery. Everything is working apart from the thumbnails.

    This is the SQL query I'm using

    Code:
    SELECT attachment.attachment_id, attachment.data_id,
                                    attachment_data.data_id, attachment_data.file_hash,                               
                                    attachment_data.thumbnail_width, attachment_data.thumbnail_height, attachment_data.upload_date,
                                    media.media_id, media.media_title, media.media_state, media.attachment_id
                                    FROM xf_attachment AS attachment
                                    INNER JOIN xf_attachment_data AS attachment_data ON
                                            (attachment_data.data_id = attachment.data_id)
                                    INNER JOIN xengallery_media AS media ON
                                            (media.attachment_id = attachment.attachment_id)
                                    WHERE attachment.content_type = "xengallery_media"
    I'm guessing I now need to change the attachment_data. I've seen that Bob replaced his whole data section in his query with this:
    Code:
    ' . XenForo_Model_Attachment::$dataColumns . '
     
  16. MattW

    MattW Member

    All working

    Code:
    SELECT attachment.attachment_id, attachment.data_id,
    ' . XenForo_Model_Attachment::$dataColumns . ',
    media.media_id, media.media_title, media.media_state, media.attachment_id
    FROM xf_attachment AS attachment
    INNER JOIN xf_attachment_data AS data ON
              (data.data_id = attachment.data_id)
    INNER JOIN xengallery_media AS media ON
              (media.attachment_id = attachment.attachment_id)
    WHERE attachment.content_type = "xengallery_media"
    ORDER BY data.upload_date DESC
    
     
  17. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    MattW: I believe the latest version of Xen Media Gallery has that fix included.

    http://xenmediagallery.com/threads/xen-media-gallery-1-1-2-released.180/
     
    MattW likes this.
    1. MattW

      MattW Member

    2. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @MattW Nicely done :rolleyes:
       
      MattW likes this.
      1
  18. simon

    simon New Member

    Hello,

    I have just purchased and installed your attachment addon. My main use is to use the ftp service into a sub domain on another server which I have set up correctly. I have a question:

    I have enabled local storage as well as FTP storage. I really would like to keep the attachments only on the ftp server rather than keeping a local copy. If at any point in the future I wanted to revert the attachments back to the default Xenforo attachment folder, how would I do this?

    Thanks in advance

    Simon
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @simon You can run the rebuild tool to move attachments between storage engines.
       
  19. simon

    simon New Member

    Hello, Ok,
    I think I understand the process now but have a few more scenarios that would like to cover.

    1)
    If I have set to store attachment on another server using ftp and on a subdomain img.mydomain.com using your addon. I now wish to change the server of the attachments using the same sub domain.

    A) Do I now need to re build the attachment catches in default xenforo system locally and then enter the new ftp server details and rebuild again

    or

    B) Backup the old ftp server data and move to new server. Then update the new FTP server location in the addon Option in admincp of xenforo I guess what I am trying to ask is what information is stored about the paths of the attachment. Is it just the sub domain or both domain and FTP Passwords. Also where is this information stored (in database?)

    2) If at anytime, I need to reimport attachments back into xenforo default attachment system to move them to a new external attachment system is it safe to delete the attachments in Xenforo default system one I have completed the 2 step process as they are no longer being served from there.

    Hope this makes sense.

    Thanks In Advance
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @simon
      For (1), you will need to do (a). The URL is stored in the database with the attachment record. It was designed this way so you can have attachment served from multiple servers at once (reading). Please note that at any given time you can only save attachment to one server (writing). If you plan to move server, I think it's possible to write a script to update the address and save everyone's time and resource.

      For (2), when you move the attachment around using the rebuild tool, the local file will be deleted unless you choose to keep a local copy. You can delete the local copy if you want to (knowing that you will need to run the rebuild tool if you want to disable/uninstall this add-on).
       
  20. simon

    simon New Member

    Are you using the same software as the Brogan Bot? :)

    I have posted twice one at 3am and the other at 9.40am and you replied to both within 10 Mins!

    Thank you very much, very nice and useful addon.

    Thanks
     
    xfrocks likes this.
Loading...