[bd] Attachment Store for XenForo 1 [Paid] 1.3.7

Trying to make sure I set this up correctly:

I'm using CloudFlare and I've successfully configured a domain attach.mydomain.com to resolve via CNAME to my S3 bucket.

In your instructions for configuring Cloud front here: https://xfrocks.com/other/articles/howto-xenforo-attachments-via-amazon-s3-and-cloudfront.60/

[HOWTO] XenForo attachments via Amazon S3 and CloudFront

Prerequisite: XenForo 1.1.4+ [bd] Attachment Store 0.9.7+ Step 1. Obtain Amazon Web Services Security Credentials + CloudFront Key Pair Follow step 1...

xfrocks.com

One of the blank fields is Domain Name and you entered the cloudfront domain name in the example.

My Attachment Options do not have an option for entering a domain name for Cloudfrount. See attached.

See attached. Does everything look OK? How can I tell that the CDN is working?

 

This is driving me crazy. I can't get images uploaded to show up in the Media Gallery. The files themselves are being uploaded into the bucket but it's not showing up in the Media Gallery itself. It seems it's not communicating with the server properly. I've tried turning off Cloud Front but no joy.

 

I disabled the addon and I'm able to add an image to a gallery. The moment I enable the addon:
IAM keys loaded
bucket name entered
local copy disabled
domain for bucket name entered (using CloudFlare)

The image will upload successfully into the S3 bucket and I can see it in bucket.mydomain.com/2017/01

But I cannot see it in my album. It won't display.

I'm getting no server log errors.

What is wrong?!

 

What's really frustrating is that I've read all your documentation. It appears that the plugin is updated so that some of the fields in the instructions are not there. I've even noticed that some of the steps other users took include options that have been removed. You speak of a config.php entry in your latest update but there are ZERO instructions on how to config.

The only way I can get images into my media gallery is to turn off your plugin.

 

As I noted, none of the screenshots look like what I'm seeing in the options.

1. Are you saying that I should leave the Domain Name field blank if I'm using Cloud Flare in the second field?
2. If I only use CloudFlare for DNS and pass through my entry (so it's not orange) then can I pass through permissions and use CloudFront.

No. The file gets uploaded into S3 but it does not appear in my Xenforo forum. It displays a broken image link.

Incidentally, it's another example of where you have made changes to the Addon but have not changed the instructions. Other respondents wrote about an option to preserve file names. That is not an option in my screenshot above.

Also, in your release notes you wrote about updates to config.php but you provided no instructions.

 

Attached are screenshots showing what it looks like after a file has been uploaded. It shows in s3 but not on the site.

Attached is the options I have. I re-enable AWS S3 uploading and left the domain field blank. I also bypassed Cloudflare so the Cloud is grayed out for my attach.domainnam.com subdomain and Cloudflare is only functioning as a DNS.

 

OK, in the hopes of helping others who may experience similar problems.

I have SSL configured for my main domain htts://www.mydomain.com.

I set up a bucket name that was a subdomain attach.mydomain.com and pointed a CNAME record to it in CloudFlare.

The issue is that you have to use Full SSL in CloudFlare in order for SSL to work properly in most cases. Since attach.mydomain.com did not have an SSL cert loaded, even though the Addon was uploading files to the S3 bucket, the files were not reachable by trying to get to https://attach.mydomain.com/2017/01/filename.jpg

I resolved the issue by simply creating a bucket without a subdomain name in it. This still allows me to use https but it resolves to an aws domain and everything is fine.

If I wanted to use a subdomain I would need to attach my own cert to it and it's more work than I wanted to do.

 

In the interest of clarity I want to make sure users of this Addon understand how to migrate their attachments to Amazon S3. The author gave the below instructions but they were confusing to me so I want to clarify after I understood what he meant.

In one place he wrote:

It's the first and second steps that were confusing to me.

Since I had already enabled S3 and Cloud Front I didn't understand what he meant by "change to use external data".

Steps:
1. Go to Options>Attachments. If you've already selected Store in Amazon S3, then you're going to have to choose Store File in External Data. Note, make sure you also select Maintain File Name. After you save the system is set up to save Attachments to the /data folder of your web root.
2. Go to Tools > Rebuild Caches and Rebuild Now under Move Attachment Data. What this does is move files from internal_data to the data directory in correct file format.
3. Copy all the folders from /data to your S3 bucket preserving all your path information. You can use a program like CloudBerry if you don't want to use AWS CLI.

After that, you can re-enable all your Amazon S3 options and go back to Rebuild Caches and Run the Storage Option Tool (using the same S3 information) to make sure that all your attachments are pointing to S3.

Hope this helps others.

 

Dumb question.

I store my attachments on external FTP. How I can change attachments URLs from http to https?

 

Looking at my xf_attachment_data table, I see that this addon is storing my S3 bucket name, access key id, and secret access key in bdattachmentstore_options for every single row. This is really wasteful for sites that store all their attachments in the same place, which I'd guess most do. It's also going to be bad news when we want to rotate our credentials, since every row in the table will need updating and we have a lot of attachments.

Is this how it's intended to work, or have I configured something wrong? Is there a way to just store the backend options just once, rather than once per attachment?

 

True, I had to modify library/XenForo/Model/Attachment.php to support the extra JOIN clause, but on the other hand it's a very straightforward change (and not too hackish, since it's very similar to how $dataColumns works). How would I propose a change like that to the XF devs?

Could you give more details on this approach?

 

I made the suggestion, but it wasn't accepted. However I was able to implement the JOIN clause without modifying core XF files, by overriding some methods in Attachment.php. There's some duplicated code, but not that much. Again, just let me know if you're interested in seeing my modifications.

So bdAttachmentStore would use the same set of credentials for every image? That would be an improvement, but not a complete solution if the other config options like bucket name and region still get duplicated in every xf_attachment_data row.

BTW, have you looked at using instance profile credentials for S3? It's the preferred way to do permissions nowadays. The AWS SDK for PHP has built-in support for that, though I don't know whether XF addons can use external dependencies.

 

A separate issue I just noticed is that the way this addon names objects in S3 buckets could cause a performance problem. Currently it uses the key naming scheme YYYY/MM/NN_XXXX, where YYYY/MM is the year and month of the upload, NN is the attachment's data_id and XXXX is the MD5 hash of the attachment data. But sequential key prefixes are bad for performance on S3. Since the YYYY/MM prefix is not really necessary, as S3 isn't a hierarchical file system and has no maximum number of files per "directory" (since there really are no directories on S3), I would recommend just using the "flat" naming scheme XXXX_NN (hash followed by data_id), to ensure attachments are uniformly distributed among S3's partitions.

 

Hello there
I get the following error after installing the plugin.

Code:

XenForo_Exception: Failed to write the attachment file. - library/XenForo/DataWriter/AttachmentData.php:146

Stack Trace
#0 /home/osxinfon/public_html/library/bdAttachmentStore/XenForo/DataWriter/AttachmentData.php(55): XenForo_DataWriter_AttachmentData->_postSave()
#1 /home/osxinfon/public_html/library/XenForo/DataWriter.php(1423): bdAttachmentStore_XenForo_DataWriter_AttachmentData->_postSave()
#2 /home/osxinfon/public_html/library/XenForo/Model/Attachment.php(539): XenForo_DataWriter->save()
#3 /home/osxinfon/public_html/library/bdAttachmentStore/XenForo/Model/Attachment.php(68): XenForo_Model_Attachment->insertUploadedAttachmentData(Object(XenForo_Upload), 1, Array)
#4 /home/osxinfon/public_html/library/XenForo/ControllerPublic/Attachment.php(205): bdAttachmentStore_XenForo_Model_Attachment->insertUploadedAttachmentData(Object(XenForo_Upload), 1)
#5 /home/osxinfon/public_html/library/XenForo/FrontController.php(351): XenForo_ControllerPublic_Attachment->actionDoUpload()
#6 /home/osxinfon/public_html/library/XenForo/FrontController.php(134): XenForo_FrontController->dispatch(Object(XenForo_RouteMatch))
#7 /home/osxinfon/public_html/index.php(13): XenForo_FrontController->run()
#8 {main}
Request State
array(3) {
  ["url"] => string(101) "https://osxinfo.net/eklentiler/do-upload.json?hash=d6ce873222a9d9f145154258b87d1aec&content_type=post"
  ["_GET"] => array(2) {
    ["hash"] => string(32) "d6ce873222a9d9f145154258b87d1aec"
    ["content_type"] => string(4) "post"
  }
  ["_POST"] => array(4) {
    ["_xfToken"] => string(8) "********"
    ["_xfNoRedirect"] => string(1) "1"
    ["_xfSessionId"] => string(32) "409687bf6cf97fc299eda3f5f1fe05b2"
    ["content_data"] => array(1) {
      ["thread_id"] => string(3) "344"
    }
  }
}

 

@xfrocks are you there?