Sunday
7
September
2008

Destination and renaming of uploaded files

Hy insterested ones,

I added for you two new improvements to Zend_File_Transfer or if you work with Forms, to the file element.

The first one is that Zend_File_Transfer allows now the usage of file filters. Note that, of course, file filters work differently to normal string filters. They do not return the content, but the location of the file, aka the real location and filename. It should also be noted that file filters will be attached AFTER the file has been received. This is due to the fact that we are not allowed to change the file before it has been received, as we would then get a attacker exception from PHP itself. Input elements, for example, will do filtering before validation. File elements will always do validation and afterwards filtering. So you could now add your own filters if you are in need. Or you wait for me to do this.

The second improvement is the first file filter.

I am happy to announce that the “Rename” filter is ready in core.
What can it do ?

Well, it will allow you to have much more influence of the behaviour of uploaded files.
Let’s see an example:

$filter = new Zend_Filter_File_Rename('C:picturesnewpics', true);

or when you use the adapter

$adapter->addFilter('Rename', array('C:picturesnewpics', true));

All uploads will now be moved to the directory C:picturesnewpics and will be overwritten if they exist.
The default is false, which means that the filter will not overwrite existing files. Beware: If the file exists, you will get a false as soon as you receive the file with an error, that the file already exists if you do not set the overwrite property.

Of course the rename adapter allows much more.
You can define the location for each single file separatly.
You can even define a new filename and extension for the uploaded file.

$adapter->addFilter('Rename', 'C:picturesnewname.txt');

For more details about supported notations look into the manual.

I hope you find this addition usefull and use it in your own projects.
PS: If you have a good idea for a additional feature to any of my components dont be shy and share your idea.

Greetings
Thomas, I18N Team Leader, Zend Framework

Back to top
  1. El_geronimo

    Tuesday, September 9, 2008 - 10:25:57

    Thank you so much for adding this! Just what I needed for my new project :)

  2. Kieran

    Tuesday, September 9, 2008 - 14:11:42

    When dealing with file uploads, I’ve found it to be desirable to store the file in an unadulterated state, so to retain the possible semantic qualities that the file name holds. Obviously, storing several files with the same name in the same directory is impossible. I solve this my storing duplicate file names in incremented sub-directories.

    eg. image.jpg would be stored as /path/to/uploads/image.jpg, however, a file of the same name, uploaded at a later time, would be stored in /path/to/uploads/0/image.jpg and so on.

    Additionally, I also disperse my uploaded files in sub-directories according to the first and (optionally) second characters of the file name. Such as: /path/to/uploads/i/m/image.jpg.

    These are just my methods, and I’ve had several discussions with colleagues as to optimal storage strategies. I suppose it leaves a lot open to question, but perhaps that is something for you to think about.

    Thanks.

  3. thomasw

    Tuesday, September 9, 2008 - 17:53:54

    Generally, creating subdirectories is no good idea. Keep in mind that Zend_File_Translate is not only working on HTTP and of course not only on the local machine.

    When we have 100 files of the same name, and have to create 100 different subdirectories it would really be a big performance problem.

    When would be possible is to define alternate targets… when the same file exists in target 1 it will be stored in target 2 (directory).

    Anyhow… Zend_File_Transfer should (and will for now) not add subdirectories at a location.

  4. Kieran

    Friday, September 12, 2008 - 20:27:50

    I don’t see why the protocol used to transfer the file to the server has any baring on creating sub directories. Surely if you have permission to rename files on the server, you also have permission to create a directory in the same location.

    Granted, when there are many files uploaded with the same name, you incur a deficit on performance. Although, this can be reduced by indexing the uploaded files and determining the destination directory of any newly uploaded files from the index.

    Having several targets sounds like a finite solution, where as if you have the ability to create directories, you would never run into the issue of overwriting files that you didn’t intend to overwrite.

    I should mention that I don’t believe this is an optimised solution, it’s just a method for handling file uploads without the need for renaming. This, I feel, is something clients (in my experience anyway) appreciate, and is quickly deployable.

    In a large scale solution, then I would want to rename the file to some unique index, and store in a directory, until that directory reaches it’s optimal capacity (dependent on the system); at which point, the storage target would change to a new directory. These files would then be cataloged in a database. I’d also (possibly) be tempted to rename the files as they are returned to the end user, according to their original filename, to retain any useful quality the filename contains (although I’ve never tried to do this).

    Thanks.

  5. Alexander

    Wednesday, September 17, 2008 - 09:23:26

    The problem with using filters is that very often a file name is formed in the action script after processing other POST variables (e.g. a new row to the db is added and the file name should contain the id of that row). In this situation the whole potential of the file transfer filters becomes completely useless…

  6. thomasw

    Thursday, September 18, 2008 - 10:02:58

    I dont get the point.

    Because you want to change the filename afterwards manually, we should not add filters at all ?
    I am working on filters which change imagesize, add watermarks, compress files and so on… and all this should be throwed away ?

    Sorry, but I think this is like break a fly on the wheel. Completly nonsense. And many people find this feature more than usefull, they are in need of it.

  7. Alexander

    Thursday, September 18, 2008 - 11:10:12

    Oh, no, that’s not what I meant!
    I am really very grateful for your work, find this filters very useful, and almost all of them announced in the issue tracker could be applied in some of my projects, so I’m looking forward to see them in action.

    But the problem is that their applicability is limited with the cases where their behaviour is independent from the other fields of the form, if I correctly understand the way they work. Probably there is a way to improve their functionality adding some kind of parameters which could be assigned during the processing of the form?

    And I want to apologize once more if I offended you… That’s absolutely not what I wanted to do.

  8. thomasw

    Thursday, September 18, 2008 - 12:14:41

    You have not offended me. There are always people which have a different opinion… this would not have stopped me to develop this new filters. :-)

    Actually, what you need is already supported.
    You can add filters and validators any time… this is not limited to the form creation. When you add your form, for example and then validate it. When the validation is done, your form is already submitted and in processing state. Now you could add new filters after validation.

    Keep in mind, or read in the manual, that file filters are applied “after the file was received” due to the fact that files can not be manipulated, but read, while they stay in the temporary cache of php.

    Of course you can also use the file filters manually everywhere in your code. See the unittests for how ot do this. But alone, they are not as comfortable as in combination with the file transfer component.

    Anyhow, this is already supported.

  9. Sam

    Saturday, September 20, 2008 - 21:58:57

    I’m looking for a way to convert the name of the uploaded file to lowercase.
    Straight in PHP that would be done by changing the $_FILES[0][’name’] value before performing move_uploaded_file to avoid another call to rename().

    Do you think something like this will be possible?

  10. Ludo

    Tuesday, September 23, 2008 - 17:35:27

    Hi,

    At first, thanks for this very necessary update of this Plugin & filter.

    But: If I’m not wrong, the $adapter requires a destination for the file via setDestination(), then if we want to move (rename, in fact) the uploaded file, it requires to use the ‘Rename’ filter, right? If this is true, then the file is always first moved to the first destination (setDestination()) then moved to the second via the filter… What if two uploads occur at the same time, with the same file names? Since setDestination() only accepts a directory, the files will override each other in the first step… If I’m not wrong…

  11. thomasw

    Tuesday, September 23, 2008 - 20:15:43

    Wrong assumption in both cases.

    You are not required to call setDestination(). If you set no destination then the system uses it’s temp path as defined by PHP.
    Zend_File_Transfer knows it and moves the file by the filter anyway.
    The reason is more a technical one. Before you have not accepted the upload you can not edit the file. And moving the file is also a sort of editing.

    Related to same time I see no problem. PHP does not save the file with it’s real name but with a internal hash. So as soon as you don’t set the same directory for both uploads and both uses the same filename in the same millisecond there is no problem.

    When this really occurs, then the second upload will fail because the file can not be moved to it’s final destination. Plain setDestination does not overwrite existing files. This feature is only available by using filters.

  12. Ludo

    Tuesday, September 23, 2008 - 22:49:25

    Thanks for the clarification! If my 1st assumption was not right, the second had no more sense.

  13. Bao

    Friday, September 26, 2008 - 00:01:13

    Hi
    I try to use this feature with Zend_Form_File_Element
    Something like this:
    $this->addElement(’file’, ‘uploadedfile’, array(
    ‘label’ => ‘upload file’,
    ‘filters’ => array(’Rename’, array($uploadDir,true));

    I got an error that say: Zend_Loader_PluginLoader_Exception: Plugin by name Rename was not found in the registry. I’m using ZendFramework 1.6.1

    Can you tell me what i did wrong? what am i missing?
    Thank you

  14. thomasw

    Friday, September 26, 2008 - 01:05:08

    First please ask in the mailing list and not in my private blog… this is only related to features and not to bug solving.

    Second you are not using Zend_Form_File_Element. I am sure by looking at your code that you are using Zend_Form instead. So better ask a Zend_Form specialist, shouldn’t you ? ;-)

  15. mat

    Monday, September 29, 2008 - 13:59:02

    Hi,

    is it possible to get Zend_File_Transfer doing the following steps?

    - uploading a _variable_ number of files (works)
    - applying validators to every file (works)
    - rename each file to a unique id (works for me somehow just for one file (the first one))
    - get a list of all files that have been renamed (doesn’t seem to work since currently not implemented exception)

    thanks in advance!

  16. thomasw

    Monday, September 29, 2008 - 15:49:08

    - yes
    - yes
    - no (issue already available)
    - yes (getFileInfo method in trunk)

  17. Tom W

    Friday, October 17, 2008 - 13:10:35

    Good work on file uploading. However, I have a really simple question:

    How do you get the file name once it has been uploaded?

  18. thomasw

    Friday, October 17, 2008 - 16:20:13

    Have you tried getFileName() ?
    I thought that the name clearly defines what it returns…

  19. brandon

    Friday, November 14, 2008 - 22:05:19

    Is there a way to apply the rename filter to change the filename but preserve the file extension without knowing what it will be? For example, i allow the upload of all types of images so the extension could be .jpg,.gif,.png, etc but I want to normalize the filenames to be image1.xxx, image2.xxx, imageN.xxx

    Thanks for your hard work.

  20. thomasw

    Friday, November 14, 2008 - 22:12:31

    For now you will have to do this manually.
    I am already working on this feature but it will not be available before the next release.

  21. Deon

    Tuesday, November 18, 2008 - 19:09:58

    Thanks for this - its what I needed. Want to move files to another directory.

    However when implementing it in 1.7.0 I had to add indexes my array when creating the filter, e.g.

    $adapter->addFilter(’Rename’, array(’source’=>’*', ‘target’=>’/new/path/’, ‘overwrite’=>true));

    Else the code as per your example would return the plain filename in the filter() function of Rename ($file = $this->_getFileName($value))
    which would mess up the rest of the functions logic.

    The source would then be what you want for your target value (the new directory location).

    Did i initialise it incorrectly or soemthing? Got it working with the code above though.

  22. thomasw

    Tuesday, November 18, 2008 - 22:44:39

    Reading this blog you would have noticed that it was written 9th September. 1.7 was released 17.November, 2 months after this blog entry. Which means that this blog entry describes the behaviour of 1.6 and not 1.7

    In the manual section there is a migration notes chapter for Zend_File_Transfer where you would have found that behaviour.

    Simply said… please read the manual :-)

  23. Marco Peverelli

    Wednesday, December 17, 2008 - 15:31:31

    You wrote: “Actually, what you need is already supported.
    You can add filters and validators any time… this is not limited to the form creation. When you add your form, for example and then validate it. When the validation is done, your form is already submitted and in processing state. Now you could add new filters after validation.”, but it doesn’t work for me…

    After the creation of the form, I run a statement like this “if($form->isValid($_POST))” and, inside the if-block, first insert other data into db, retrieving auto-incrementing key, then try to add a filter so $form->getElement(’profilepic’)->addFilter(’Rename’,'picUser’.$id)

    Could you kindly explain me what I’m doing wrong?

    Thanks,
    M.

  24. thomasw

    Wednesday, December 17, 2008 - 16:15:29

    Not really because I’ve answered to Alexander before. :-)

    There have been 2 or 3 mails in past on the mailinglist from people doing exactly the strategy I’ve described.

    Also it depends on the used release.

    Be aware that you don’t receive the files and add the filter afterwards. You can check this by using isReceived().
    Because some methods receive in background. getValues() is an example.

    Also to mention: This article describes Zend_File_Transfer… I don’t know if the described solution works also on Zend_Form.

    Beside that I would say that you should ask in the mailinglists or on IRC #zftalk because I can’t and also won’t solve bugs within my blog as you may understand. :-)

  25. Marco Peverelli

    Wednesday, December 17, 2008 - 18:25:40

    Sure, I understand. But I didn’t mean that behavior to be a bug in your code, I mean the problem is my lack of knowledge! ;)

    BTW, which mailing list are you talking about? So that I can search and learn…

    Thanks again,
    M.

  26. thomasw

    Wednesday, December 17, 2008 - 18:53:36

    Look here http://framework.zend.com/community/overview for the mailing lists. Or use http://nabble.com to search them online.

  27. Ben

    Monday, February 2, 2009 - 18:21:43

    This is a great filter, but I am having difficulty extending it to meet my needs.

    I understand that it can rename and overwrite files, but that functionality isn’t considerate in the real world of users where many people using a CMS might have an image for example named “test.jpg” and these uploaded images all upload to a single directory “images”.

    For example, if a user adds his image “test.jpg”, the next time an image of “test.jpg” is uploaded, the original is overwritten. This will not do as users will not expect their image to get overwritten.

    What means is there of extending this to dynamically increment the filename by 1 in the case of a file already existing on the server? e.g. test-1.jpg if test.jpg exists?

    Thanks.

  28. thomasw

    Monday, February 2, 2009 - 21:22:35

    When you use a CMS with a huge amount of users and also use only one directory you will soon have other problems than only overwritten files. You should never use a single directory for a CMS for uploaded files. I don’t expect that your filesystem can handle 100.000 files in one directory really fast.

    Beside of this, it makes no sense to mock about things which are already promoted but not integrated for now. Existing feature requests are no bugs, even if many people have a different opinion on this :-)

    At last, when you speak of extending this is something which you can do yourself. Any filter can be extended for own purposes and be attached to the file component. When you are in need of a file-x syntax simply extend the core class and write your own extension of it.

    Btw: Even with the actual available methods and classes is a solution for this case possible without extending something.

  29. Ben

    Monday, February 2, 2009 - 23:55:10

    Hi there,

    Thanks for the fast response.

    > You should never use a single directory for a CMS for uploaded files.

    I agree, but I have to work within the constraints of a system I am given.

    >it makes no sense to mock about things which are already promoted but not integrated for now.

    I wasn’t mocking anything. What features are there that are promoted but are yet to be implemented?

    > When you are in need of a file-x syntax simply extend the core class and write your own extension of it.

    I hadn’t thought of doing it this way, but I will look into it now.

    Thanks again for your replies. I hope you don’t take my comments as criticism and they aren’t intended as bug reports. Other than meeting the need of this specific job, I’ve been very happy with the filter!

  30. thomasw

    Tuesday, February 3, 2009 - 00:42:17

    I knew how you meant it and this question and related were already asked and answered by me in past.

    Please keep in mind that I am no employee of Zend and I do this in my sparetime, so I need some days.

  31. thomasw

    Tuesday, February 3, 2009 - 01:11:52

    Btw: You can add missing features also at ZF’s issuelist: framework.zend.com/issues :-)