Serious help: [bd] Attachment Store

Discussion in 'Other' started by vitoreis, Feb 22, 2016.

  1. vitoreis

    vitoreis New Member

    #1 , Feb 22, 2016
    Last edited: Feb 22, 2016
    Hi,

    I'm in serious problems with attachments at one board. This board his 10 years old an we can not see attachments older than two months.

    Scenario:
    1. Have [bd] Attachment Store working well with S3/CloudFront from many mouths.
    2. Decide to came back to default xenforo structure because we have problems with attachments in Showcase addon and Classifieds addon.
    3. Options > Attachments > [bd] Attachment Store: Default
    4. Tools > Rebuild Caches > Update Attachment Data Storage Options > Mode: Default (I know, was an error. I must run "Move Attachment Data" :()
    5.
    attachments1.png

    Link to image: site.com/forum/index.php?attachments/10426665_812568642132968_3309731536130555255_n-jpg.323122/

    I can move images from last two mouths to and from S3 but older images never change.

    I believe that original images still survive at S3 but the script can't bring it back to forum.

    I'm with serious problems... it's an old board with thousands of images and active members. any help?

    Thanks
     
  2. xfrocks

    xfrocks XenForo rocks!
    Staff Member

    So the only problem is incorrect attachment data in db? Can you do a db restore? Otherwise, I could write a script to restore the metadata I guess...
     
  3. vitoreis

    vitoreis New Member

    No, can not do a db restore because i get a lot of daily posts. I will lose a lot of information.
    If this is possible... will gone be amazing!

    I can move back all new (2 mouths) attachments to S3 and run that script to restore the metadata - images from S3 because they are not at my server but are at Amazon S3
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @vitoreis Can you take some records from xf_attachment_data for me? I need to look to make sure I understand it right. Please start a conversation with me with the db dump. Around 200 records would be great.
       
    2. vitoreis

      vitoreis New Member

      vitoreis @xfrocks Send by conversation! Starting moving attachments at my server to S3!

      THANKS!
       
  4. vitoreis

    vitoreis New Member

    Finish rebuild attachments. They are now served by Amazon S3 - but all attachments uploaded before 15-Dez-2015 have the wrong URL with error because they don´t exist in my server.

    Really need that script to restore the metadata of attachments that exist in S3 but have incorrect attachment data in db.
     
  5. vitoreis

    vitoreis New Member

    Maybe same member can help me? How to "force" Amazon S3 «path/image_name» in my wrong database?
     
    1. xfrocks

      xfrocks XenForo rocks!
      Staff Member

      xfrocks @vitoreis I will send you a custom script for this via conversation.
       
    2. vitoreis

      vitoreis New Member

      vitoreis @xfrocks Don't receive nothing :(

      Can you please read your Inbox?

      Thanks,
      Vitor
       
  6. vitoreis

    vitoreis New Member

    I'm so frustrated with this... I wondered if anyone else has some help to recreate default xenforo attachments structure?
     
  7. rsi

    rsi New Member

    I'm having the same problem, the guy never responds when you ask for help.
     
  8. vitoreis

    vitoreis New Member

    I think we can change "StorageOptions.php" to ignore if file exists and rename it at "xf_attachment_data" table...

    If only I could program in PHP :( The solution is somewhere around here ...

    PHP:
    <?php

    class bdAttachmentStore_CacheRebuilder_StorageOptions extends XenForo_CacheRebuilder_Abstract
    {
        public function 
    getRebuildMessage()
        {
            return new 
    XenForo_Phrase('attachments');
        }

        public function 
    showExitLink()
        {
            return 
    true;
        }

        public function 
    rebuild($position 0, array &$options = array(), &$detailedMessage '')
        {
            
    $options array_merge(array(
                
    'batch' => 50,
                
    'fileExists' => false
            
    ), $options);

            
    $db XenForo_Application::getDb();

            
    /* @var $attachmentModel XenForo_Model_Attachment */
            
    $attachmentModel XenForo_Model::create('XenForo_Model_Attachment');
            
    /* @var $fileModel bdAttachmentStore_Model_File */
            
    $fileModel XenForo_Model::create('bdAttachmentStore_Model_File');

            if (!isset(
    $options['maxDataId'])) {
                
    $options['maxDataId'] = $db->fetchOne('SELECT MAX(data_id) FROM xf_attachment_data');
            }

            if (
    $position >= $options['maxDataId']) {
                return 
    true;
            }

            
    $attachmentDatas $db->fetchAll('
                    SELECT attachment_data.*
                    FROM `xf_attachment_data` AS attachment_data
                    WHERE attachment_data.data_id > ?
                        AND attachment_data.data_id <= ?
                        AND ' 
    . ($options['old']['mode'] != 'attachment' ? ('attachment_data.bdattachmentstore_engine = ' $db->quote($options['old']['mode'])) : '(attachment_data.bdattachmentstore_engine IS NULL OR attachment_data.bdattachmentstore_engine = "")') . '
                    ORDER BY attachment_data.data_id ASC
                    '
    , array(
                
    $position,
                (
    $position $options['batch'])
            ));
            
    $position += $options['batch'];

            
    $defaultEngine $fileModel->getDefaultEngine();

            if (
    $defaultEngine) {
                
    $defaultEngineOptions $fileModel->getStorageOptions($defaultEngine);
            } else {
                
    $defaultEngineOptions = array();
            }

            foreach (
    $attachmentDatas AS $attachmentData) {
                
    $oldStorageOptions = @unserialize($attachmentData['bdattachmentstore_options']);
                if (!
    is_array($oldStorageOptions)) {
                    
    $oldStorageOptions = array();
                }
                
    $newStorageOptions $oldStorageOptions;

                
    $oldMatches true;
                switch (
    $options['old']['mode']) {
                    case 
    's3':
                    case 
    'ftp':
                        if (empty(
    $options['old'][$options['old']['mode']])) {
                            
    // s3 and ftp needs their options...
                            
    $oldMatches false;
                        } else {
                            foreach (
    $options['old'][$options['old']['mode']] as $oldKey => $oldValue) {
                                if (empty(
    $oldValue)) {
                                    
    // nothing to check against
                                    
    continue;
                                }

                                if (!isset(
    $oldStorageOptions[$oldKey])) {
                                    
    $oldMatches false;
                                } elseif (
    $oldStorageOptions[$oldKey] != $oldValue) {
                                    if (
    $options['old']['mode'] == 'ftp' AND $oldKey == 'port' AND $oldValue == 21 AND empty($oldStorageOptions[$oldKey])) {
                                        
    // for ftp, empty port means 21
                                    
    } else {
                                        
    $oldMatches false;
                                    }
                                }

                                if (
    $oldMatches) {
                                    
    // matches!
                                    
    if (isset($defaultEngineOptions[$oldKey])) {
                                        
    $newStorageOptions[sprintf('%s_%d'$oldKeyXenForo_Application::$time)] = $newStorageOptions[$oldKey];
                                        
    $newStorageOptions[$oldKey] = $defaultEngineOptions[$oldKey];
                                    }
                                }
                            }
                        }
                        break;
                }

                if (!
    $oldMatches) {
                    
    // old storage options do not match, ignore
                    
    continue;
                }

                if (
    $defaultEngine != $attachmentData['bdattachmentstore_engine']) {
                    
    $newStorageOptions[sprintf('mode_%d'XenForo_Application::$time)] = $attachmentData['bdattachmentstore_engine'];
                }
                foreach (
    $defaultEngineOptions as $defaultKey => $defaultValue) {
                    if (!isset(
    $newStorageOptions[$defaultKey])) {
                        
    $newStorageOptions[sprintf('%s_%d'$defaultKeyXenForo_Application::$time)] = null;
                    }

                    
    $newStorageOptions[$defaultKey] = $defaultValue;
                }

                if (!empty(
    $options['fileExists'])) {
                    
    $fakeAttachmentData $attachmentData;
                    
    $fakeAttachmentData['bdattachmentstore_engine'] = '';
                    
    $filePath $attachmentModel->getAttachmentDataFilePath($fakeAttachmentData);
                    
    $fileUrl null;

                    if (
    $defaultEngine) {
                        
    $ourFilePath $fileModel->getAttachmentDataFilePath($fakeAttachmentData);
                        
    $fileUrl $fileModel->getFileUrl($defaultEngine$newStorageOptions$ourFilePath);
                        
    $fileExists = !!@file_get_contents($fileUrl);
                    } else {
                        
    $fileExists file_exists($filePath);
                    }

                    if (!
    $fileExists) {
                        
    // file cannot be found
                        
    XenForo_Helper_File::log(__CLASS__sprintf('File not found: %s'$fileUrl $fileUrl $filePath));
                        continue;
                    }
                }

                
    $dw XenForo_DataWriter::create('XenForo_DataWriter_AttachmentData');
                
    $dw->setExistingData($attachmentDatatrue);
                
    $dw->set('bdattachmentstore_engine'$defaultEngine);
                
    $dw->set('bdattachmentstore_options'$newStorageOptions);
                
    $dw->save();
            }

            
    $detailedMessage XenForo_Locale::numberFormat($position);

            return 
    $position;
        }

    }
     
  9. vitoreis

    vitoreis New Member

    So simple?
    phpMyAdmin:
    Code:
    UPDATE  `xf_attachment_data` SET  `bdattachmentstore_engine` =  's3'
     
  10. vitoreis

    vitoreis New Member

    OK, I get it with two querys:

    Code:
    1. UPDATE `xf_attachment_data` SET `bdattachmentstore_engine` = 's3'
    
    2. UPDATE `xf_attachment_data` SET `bdattachmentstore_options` = 'MY_BYN_CODE' WHERE `bdattachmentstore_options` IS NULL
     
  11. rsi

    rsi New Member

    What did those queries do exactly?

    The problem I have is when I move my board (whole website) to another server and try to run the Attachment Thumbnail Rebuild tool that is built in to xenforo. I get thousands of errors because all of those files were uploaded with my old username, password, and I think even the old host name and the files were stamped as such. On a new server my ftp username and all of that changed and causes a big problem. Obviously the guy never thought about that being a problem.

    I would like to be able to remove this add-on completely and put everything back to xenforo default (file structure and all) if the guy cannot update the add-on again to resolve these issues. I would be nice to get some support here.

    Again, what exactly do those queries do and what did they correct?

    Thanks…
    .
     
  12. vitoreis

    vitoreis New Member

    They restore the metadata in my db. I have all attachments stored at S3 but (some) incorrect attachment data in db!

    Now, that I have all files being served from S3+CloudFront, I will try to get it back to xenforo default.

    The steps by @xfrocks :
    • Change mode to Default
    • Then run the rebuild tool: Move Data (not the Update Storage Options one)
    Let's see if it works!
     
  13. vitoreis

    vitoreis New Member

    Can not get it back to xenforo default! After use the rebuild tool (Move Data) have a lot of attachments with the link to amazon bucket :mad:

    @xfrocks , some help here are very appreciated!
     
Loading...