Monday, 4 October 2010

Bulk upload files with metadata into SharePoint using PowerShell: UPDATE

This is an update to an article that I wrote back in July, which contained the first version of a PowerShell script to copy multiple documents from a file system into a SharePoint 2010 document library, automatically tagging them with column values (metadata) during upload. Back then, the script had very limited functionality, but I have now added some new features, converted the script to a function, and uploaded the PS1 script file with documentation to Codeplex for download.

In summary, the script now includes the following features:

- Upload multiple files from a local folder or network share into SharePoint 2010 document libraries
- Files can be uploaded from a single folder or an entire folder hierarchy
- Retain the folder hierarchy from the source path into the destination document library
- Choose the root or a specific destination folder in the document library to copy the files into
- Support for adding values to the following column types in the manifest file:
      - Single line of text
      - Multiple lines of text
      - Choice – Single value (drop-down/radio button)
      - Choice – Multiple value (checkboxes)
      - Currency
      - Number
      - Yes/No (checkbox)
      - Person or Group – Single value
      - Person or Group – Multiple value
      - Hyperlink or Picture
      - Managed Metadata – Single value
      - Managed Metadata – Multiple value
- Automatic file check in after upload, if required
- Automatic file approval after upload, if required
- Option to overwrite existing files in the destination document library
- Option to flatten the structure when copying from a multi-level folder hierarchy to a single folder in a document library (i.e., merging sub-folders to a single location)

copyfilestosp_screenshot

You can download the script and user guide from Codeplex at http://spfileupload.codeplex.com. One of the advantages of using PowerShell for this tool rather than managed code, is the ability for you to customise the script should you wish to add any custom features as required. I intend to continue adding extra features in the future, but also welcome any ideas or contributions to the Codeplex site if you have them.

94 comments:

  1. Hi Phillip, is the "Creator"a and "Modifier" information transfered from the file to sharepoint properties with your script?

    ReplyDelete
  2. Not yet - I am planning to grab the author and editor fields from Office documents in a future version. I had a look at the possibilities of getting user information from non-Office files, but couldn't find any obvious way of doing it on first glance.

    It's on the list!

    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
  3. Hi Phil! This is great.. I would really like to see more of this. I tried your script, and it works like a charm! I am using it to place metadata that is already stored in the 'Term Store' to multiple documents on upload. The one problem I can't figure out is, how can I give a document a metadata from level 2 or 3?

    Example;
    Metadata: World - Continent - Country

    I want to add the metadata "Country" to a document. Any suggestions?

    Thx in advance =)
    A.

    ReplyDelete
  4. Hi "A.",

    It doesn't matter which level the term exists in the term store, you just enter the value in the XML file the same way. In your example above, just add "Country" as the term and the script will find it on level 3.

    Phil

    ReplyDelete
  5. Don't know what went wrong the first time, but then I'll give it a second try =) Thanks for all help Phil!

    ReplyDelete
  6. Just to be sure, I'm assuming your "Country" has already been created in the term store? If not, the script will create it for you, but only on Level 1 as it won't know where in the structure to put it.

    ReplyDelete
  7. That is correct, it has already been created.. I haven't had the time to test it out yet, but will do it soon.. I'll let you know =)

    ReplyDelete
  8. Hi Phil.. I'm reading the userguide about replicating an entire folder hierarchy to a library. Is it possible to add different kind of metadata to these documents at the same time?
    thanks

    ReplyDelete
  9. Not at the moment. I mentioned this in my first article and would require a change to the script that looked at a file plan to upload folders instead of picking from a folder directly. Not too difficult to do, and I'll add it to the list for future changes.

    ReplyDelete
  10. Hi Phil, congratulations for your tools, i'm testing this tool in Sharepoint Foundation 2010, can work in this scenario ?
    I pass this command:
    Copy-FilestoSP -LocalPath "C:\Temp" -SiteUrl "http://w2k8/Id" -Library "Documentos compartidos" -ManifestFilePath "C:\Sharepoint\Manifest.xml" -Overwrite
    I receive this error into powershell :
    No se puede llamar a un método en una expresión con valor NULL.
    En C:\Sharepoint\Get-SPScripts.Copy-FilesToSP.ps1: 331 Carácter: 21
    + $web.Dispose <<<< ()
    + CategoryInfo : InvalidOperation: (Dispose:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

    ReplyDelete
  11. Difficult for me to diagnose as the error message you are getting is not in English, but it looks like the $web variable might be NULL - which means that the SiteUrl you are entering may not exist or you do not have permissions to it in PowerShell. Try the following commands:

    $web = Get-SPWeb http://w2k8/Id
    $web

    If nothing appears after you type the second line and hit ENTER then there is a problem accessing the site in PowerShell

    ReplyDelete
  12. Hi already Phil, the command Get-SpWeb http.... returns as object not found.May be this function or variable don't exists in Sharepoint Foundation ?

    ReplyDelete
  13. Phil, don't worry, i found my problem, thanks a lot.If i run my PowerShell with Import Modules all work fine!! I'm very happy, now i try to run your script and function across a normal CMD to make a BAT file.

    ReplyDelete
  14. Hi Phil,

    This looks like a great product! I only have one question, is there any plans for supporting the column type Date?

    Best regards
    Morten

    ReplyDelete
  15. Hi Phil,

    I have an additional question; I have a folder of about 2000 files, that is to be put into one List in SharePoint. Most of these documents have the need for separate manifest XML files. (I'm moving an old Archive from Lotus to SharePoint so the making of the XML files will be automated).

    Is there any way that I can automate the process of running the Copy-FilestoSP command for the different files? If not the process will take to long.

    Best regards
    Morten

    ReplyDelete
  16. Morten,

    Re: Date/Time column - there are plans, but couldn't give you an idea when due to lack of available time at the moment

    Re: multiple manifest files - the only way I have thought of doing this in the past was creating a file plan - possibly from a CSV file - and then importing this using a PowerShell script that in turn will run the Copy-FilestoSP command multiple times, specifying the correct manifest file for the correct files/folders. Again, all time consuming stuff....

    ReplyDelete
  17. Hi Phil,

    Can this script use on the SharePoint 2007 with Powershell 2.0?

    //i'm newbie to PS and MOSS.

    ReplyDelete
  18. It can't be used in it's current form, but can be adapted for use on MOSS - I guess if you're a newbie to PS and MOSS then that's not a lot of help. It's on the list of things to do...

    ReplyDelete
  19. Hi Phil, Great Script!

    Does it only work with the default manage MetaData Service? I created a new MetaData service (even if I make that one 'Default') but I cant get the Script to add MetaData from that service. If I try to add from the default it works. Have also tried to deleted the default Service from the Association list, but then I don't even get the blue 'Setting Value on Column....'

    Another thing is when I import a dokument with Swedish special character I get error.

    Best Regards, Peter

    ReplyDelete
  20. Phil,

    I am very grateful for the work you have done with this function. However, I do have a question/problem. I have to update two columns of metadata in my doc library for several thousand files. The files are segmented into folders on my system based on the value that needs to be put into the metadata column in my library. So I upload one folder just fine, but when I go to the next folder that uploads some new files and some existing files but with different metadata for a different column, I get an error that says: "The script has stopped because there hiss been an error." How can I tell what has caused the error? I can delete the files from the library and upload from the second folder just fine. But when I go back to the first and hit a duplicate file, it fails again. I am using the -Approve and -CheckIn options. When I remove these options, I don't get an error, but I don't see the files in my library either. Please help!

    Thanks, Jamie

    ReplyDelete
  21. Peter and Jamie - unfortunately, I will not be able to investigate your exact issues as I will need to rig them up, which I don't have time to do.

    However, to Jamie in particular, you should be able to find the error causing your problem by changing the catch block to the following:

    catch
    {
    write-host "The script has stopped because of the error: "$_
    }

    I realised by looking at it today that I formatted the catch block incorrectly and will update it in a future version of the script.

    Phil

    ReplyDelete
  22. Phil,

    Thanks for you quick response. I changed the catch as instructed and the additional error message says: "The script has stopped because there has been an error. Exception calling "Add" with "3" argument(s): "The file "http://xxxx/sites/xxxx/Documents/xxx.pdf" is not checked out. You must first check out this document before making changes." .

    (By the way, I am trying to add three different values to the metadata column through the xml file.)

    Is there a checkout command I should add to the script?

    Thanks,
    Jamie

    ReplyDelete
  23. Phil,

    Thanks for your quick response. I did as instructed and got this error message: "The script has stopped because there has been an error. Exception calling "Add" with "3" argument(s): "The file "http://xxx/Documents/xxxx.pdf" is not checked out. You must first check out this document before making changes." ."

    Is there somewhere I should or could put a command to checkout the document before changing it and then check it back in?

    Thanks,
    Jamie

    ReplyDelete
  24. Jamie,

    This is from memory, so may not work without testing, but you could try the following after the $spFullPath = $folder.Url + $spFilePath line:

    $existingFile = $web.GetFile($spFullPath)
    if (($existingFile.Exists) -and ($existingFile.SPCheckOutType -eq "None"))
    {
    $existingFile.CheckOut()
    }

    ReplyDelete
  25. Hi Phill,
    I need to create a 60000 Document Set with different-different properties in a single document Lib.
    Also each Document Set contain some document with metadata properties like country and other document property.

    What you suggest, shell I use the PowerShell script (http://spfileupload.codeplex.com/) or do something different.

    Awaiting your reply

    ReplyDelete
  26. Phool - you may be able to use elements of the script with Document Sets, but not the whole thing. I haven't done a lot of work yet with PowerShell and Document Sets, but it's something I will look into at some point.

    ReplyDelete
  27. Thanks Phil...
    I will try with elements, then I will share with you.
    Once again thanks....

    ReplyDelete
  28. Phil,

    I have finished modifying your script to utilize a .csv file with different metadata information for each file. It only modifies managed metadata columns, but it may be useful for some of your readers. How do I go about getting the modified script to you?

    Thanks,
    Jamie

    ReplyDelete
  29. Sounds great Jamie, thanks. You can send it by e-mail to phillip.childs@gmail.com

    ReplyDelete
  30. Hi Phil,
    The script looks great! I successfully uploaded and updated metadata ... but with taxonomy is is not working with me. I've created simple term (A,B,C). Created manifest.xml:
    ===




    a




    sample description



    ===

    this is the command:
    Copy-FilestoSP -LocalPath "C:\install\docs" -SiteUrl "http://portal.aes.local" -Library "documents" -ManifestFilePath "C:\install\Manifest.xml" -IncludeSubFolders –Overwrite

    and here you can see that description field is successfully updated but taxonomy NOT:
    Copying Balance of Plant.Buildings.Acts.1construction.txt to Documents in AES BG
    Document Management System ...
    Setting value on column Description ...
    Value set to sample description

    Copying Balance of Plant.Buildings.Acts.1installation.txt to Documents in AES BG
    Document Management System ...
    Setting value on column Description ...
    Value set to sample description

    Am I missing something ???

    ReplyDelete
  31. Hey Phil,
    did you get from Jamie the modified script that can read a CSV file to describe each file with Managed Matadata?

    This would be very useful to me.
    It would be great if you share.

    Thanks,
    Tony

    ReplyDelete
  32. Hi Phil,
    Great script - works with managed metadata!
    ... but...
    Here is a question/foible/quirk for you...

    When files with the attached managed metadata imported in to the drop-off library using your script, the rules set on the drop-off library wouldn't work on the imported files...? The files wouldn't move out of the drop-off library.

    Do you have any ideas why it's happening?

    Thanks,
    Vadim

    ReplyDelete
  33. Sorry Vadim, No ideas without testing it for myself.

    ReplyDelete
  34. Hi Phil,
    Please let me know how you get on...
    It seems to me that a rule 'tries' to move file BEFORE the managed metadata is attached to the file from the script - the file exists but no metadata attached yet. Therefore SharePoint 'marks' the file as 'no metadata attached' and leave it for dead...

    Regards,
    Vadim

    ReplyDelete
  35. The column type date

    send date as string in format RRRR-MM-DD

    for example: (remove #)
    <#Column name="MyDate" type="Text">
    <#Values>
    <#Value>2010-12-31
    <#/Values>

    ReplyDelete
  36. Sorry I wrote format in czech language
    Correct format is YYYY-MM-DD

    ReplyDelete
  37. hi Phil,

    at first nice work!
    But after the installation I get some errors in the Powershell, when I run all the scenarios in the commandline with my server setting

    You cannot call a method on a null-valued expression.
    At C:\Install\Get-SPScripts-Copy-FilesToSP.ps1:331 char:21
    + $web.Dispose <<<< ()
    + CategoryInfo : InvalidOperation: (Dispose:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

    I hope you can help me...

    Best regards

    ReplyDelete
  38. Hi Phil,

    is there a solution for a automation with a scheduled task or with the "File Server Resource Manager - File Management Task on MS Server 2008?

    ReplyDelete
  39. is there a way to set the owner of the document through the manifest.xml file? we are looking at using this script to migrate files from our existing DMS, and maintaining information about ownership would be very helpful

    ReplyDelete
  40. Avidan,

    It depends on the document format and where the ownership information is stored. I have experiented with grabbing metadata from Office documents in PowerShell, but not direct from a file share.

    ReplyDelete
  41. Is there a what to grab part of the file name and write that out to a xml file so what it uploads into sharepoint it could have the meta data of a file name exp:032120012214jdoe.wav it would grab 03212011 put in field date. Then grab 2214 put in field ext. Than grab jdoe and write to user.

    ReplyDelete
  42. Sam, you can use regular expressions to achieve this in PowerShell - take a look at this excellent article on the subject: http://powershell.com/cs/blogs/ebook/archive/2009/03/30/chapter-13-text-and-regular-expressions.aspx

    ReplyDelete
  43. Cool script, but i'm gettting:



    Adding folder 11 S&P to Shared Documents in 033 Twinridge ...
    The script has stopped because there has been an error.


    I'm using:

    PS C:\Users\Administrator> copy-filestosp -LocalPath "E:\033 - Twinridge" -SiteURL http://abc/team_c/033 -Library "Shared documents" -IncludeSubFolders -overwrite

    I've updated your script with:

    catch {write-host "The script has stopped because of the error: "$_}

    Yet it still doesn't show the error message....

    How dO i force it to show the error?

    Thanks!

    Thomas

    ReplyDelete
  44. Hi Phil,

    This is exactly what we need but I am unable to create a manifest.xml file. I am new so can you advise the best way to create a manifest file to use with this?

    Thanks

    Srini

    ReplyDelete
  45. Hi Srini - there is an example Manifest XML file on the Codeplex site

    Phil

    ReplyDelete
  46. Phil
    Great script :)

    How would I package the script to run in a batch file for a scheduled task?

    thanks,
    --samuel

    ReplyDelete
  47. Hi Samuel - you wouldn't :)

    Run the script directly as a scheduled task instead:

    http://get-spscripts.com/2011/01/running-sharepoint-powershell-script.html

    Phil

    ReplyDelete
  48. Hi Phil,
    Thanks for this !
    But i'm still having a simple problem.
    My site's url is :
    http://doc.toto.org/team/it
    My library's url is :
    http://doc.toto.org/team/it/library/projects

    It work great if my library is at the root of the the collection site. ie :
    http://doc.toto.org/team/it/projects_test
    But with the sub site library it won't. Any idea ?
    PS : I use the site model "Document center"
    PS2: sorry for my english...

    ReplyDelete
  49. I found my mystake. I use the name of the library note the title...

    Sorry...

    ReplyDelete
  50. This script is great but it will require that I create a XML file per document. In my case, the values of the metadata are different for each file and I have thousand of files. I have all the medata values in a CSV file. Can you please point me into the right direction?

    Thanks!

    ReplyDelete
  51. How do I run this WITHOUT the Manifest file? I dont need this but the script wont let me run without specifying???

    ReplyDelete
  52. Hi Phil,

    thx a lot for this script.
    I have a question, is it possible to write in the metadatas of an "sharepoint file" the metadatas of the original file ?
    Like "title", "keywords" in rightClik/property/details/ of a Word file

    PS: Sorry for my english :)

    ReplyDelete
  53. Hello Phil, does this work with https ?

    i tried the script on different machines but keep getting this.

    Henk.

    You cannot call a method on a null-valued expression.
    At C:\Get-SPScripts.Copy-FilesToSP.ps1:331 char:21
    + $web.Dispose <<<< ()
    + CategoryInfo : InvalidOperation: (Dispose:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

    ReplyDelete
  54. Hi Phil,

    Does your script now supports Date/Time? Really cool PS script but I have about 6 date/time fields that need to be imported as well.

    Amber

    ReplyDelete
  55. I have tried running the script but get the error below which appears to be related to authentication. Are there any authentication settings that need to be set?

    You cannot call a method on a null-valued expression.
    At C:\Install\Get-SPScripts-Copy-FilesToSP.ps1:331 char:21
    + $web.Dispose <<<< ()
    + CategoryInfo : InvalidOperation: (Dispose:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

    ReplyDelete
  56. Quick Question, is there a way to generate the XML files used for thsi script using a CSV file in powershell, I know about the import-csv function, I would mostly be looking just to do the column names and values, but the metatdate field type would be nice as well.

    ReplyDelete
  57. Phil...Is there a way to set a simple column value from the command line?? I just need to set the document "Title" to a unique ID so I can update the rest of the columns programmatically. Make sense? I want to avoid a painful process of generating manifest files for each of my 10k files I have to import.
    Thanks.

    ReplyDelete
  58. great script.
    FYI: Can't be run in a remote shell.
    to make it even better have the XML to reference to a file would be great (so not one static XML for one target folder) and date column.
    If i get it working will try to upload it.

    cheers

    ReplyDelete
  59. I have your script working great for the most part but it does not work with the managed metadata and am not sure why? Everything seems correct.

    ReplyDelete
  60. Hi Phil, i've a problem when i try to execute the command Copy-FilesToSp. Powershell says me that the term is not recognized as the name of a cmdlet...
    What i've to do?

    Tnx a lot

    ReplyDelete
  61. Hi Phil, are there any changes to be made when copying to a document set in a document center as opposed to a doc library?

    ReplyDelete
  62. Hi Jason. Unfortunately, I haven't tested it in that way, so if you do, please let me know ;-)

    ReplyDelete
  63. Hi Phil,

    Nice article and thanks for sharing. Do you have updated code to support date/time type column?

    Thanks
    Amber

    ReplyDelete
  64. Hi Amber - unfortunately I don't have an update yet, but you should be able to modify the script so that a date/time column can be entered as a string value

    ReplyDelete
  65. Hi Phil,

    All metadata works except managed metadata

    The script doesnt show it being applied when running

    ReplyDelete
  66. This comment has been removed by the author.

    ReplyDelete
  67. Hi,
    great its working even with Managed Metadata. I've similar issue with setting the managed metadata, but it can be solved if you set the default Site Collection Term Store on the Central Admin or don't use $termStore = $taxonomySession.DefaultSiteCollectionTermStore but instead of this use $termStore = $taxonomySession.DefaultKeywordsTermStore in the script.

    The DefaultKeywordsTermStore is automatically setup after installation of MMS.
    Hope this help.

    The modification to use CSV as an input file for setting metadata is welcome.

    ReplyDelete
  68. The script works perfectly! :-)

    Only 2 problems are that when I put in a (valid) UserMulti (DOMAIN\USER) it doesn't appear in SharePoint and there is no error message from the script - and then when uploading a word-doc it won't get recognized as such in SharePoint!?

    ...any clues out there?

    ReplyDelete
  69. Amazing work Phil! and thanks for sharing it. I have one question. I have more than 100,000 documents to copy from local folder to sharepoint doc library where each document has separate xml metadata.How do i manage the metadata in batch?
    Appreciate your help.

    ReplyDelete
  70. Hi Phil,

    Have you considered adding some regex/replace to capture the filesystem naming of folders and files with special characters?

    Cheers and great work
    Mike

    ReplyDelete
  71. Can be controlled the upload permission we want to upload folder + file to every user but ONLY can see the folder/file he owns. Regards.

    ReplyDelete
  72. Hi,
    Maybe posting a sample of what the meta-data is expected to look like would be helpful. It might also, help answer a lot of people's questions.
    You have written a great script, and your reward is a gazillion questions :-)
    Thanks for what you have provided. It is great.
    Sincerely,
    Skip

    ReplyDelete
  73. Hi Phil,

    thanks for this create Script. How can i send you my Version which updates the Modified Column in Sharepoint?

    Reinhard

    ReplyDelete
  74. Hi Phil,

    I'm getting the following error when I run the script:
    You cannot call a method on a null-valued expression.
    At C:\Install\Get-SPScripts-Copy-FilesToSP.ps1:331 char:21
    + $web.Dispose <<<< ()
    + CategoryInfo : InvalidOperation: (Dispose:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull

    What am I doing wrong?

    ReplyDelete
    Replies
    1. Hi,

      any solution to the problem reported by the user ranndomista?

      Delete
  75. Hi Phil,

    I followed the directions, but get a "The term 'Copy-FilestoSP' is not recognized as the name of a cmdlet, function...

    Am I missing something obvious?

    ReplyDelete
  76. Hi Phil,

    Do you have a sample script that reads from csv? We're looking to migrate about 400K worth of physical files and each has a different value on its properties. Hoping you have one handy for us to start with.

    Thanks,
    Ellen

    ReplyDelete
  77. Hi Phil i am getting following err:

    PS C:\upload files to SP> .\Get-SPScripts.Copy-FilesToSP.ps1
    SharePoint 2010 Multiple File Upload Script - Get-SPScripts.com
    PS C:\upload files to SP> Copy-FilesToSP -LocalPath "c:\Test" -SiteUrl "http://spn10:10630/sites/newsp/demo" -Library "
    hared Documnets" -ManifestFilePath "C:\Install\DocManifest.xml" -IncludeSubFolders -Overwrite
    The term 'Copy-FilesToSP' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
    the spelling of the name, or if a path was included, verify that the path is correct and try again.
    At line:1 char:15
    + Copy-FilesToSP <<<< -LocalPath "c:\Test" -SiteUrl "http://spn10:10630/sites/newsp/demo" -Library "shared Documnets"
    -ManifestFilePath "C:\Install\DocManifest.xml" -IncludeSubFolders -Overwrite
    + CategoryInfo : ObjectNotFound: (Copy-FilesToSP:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

    PS C:\upload files to SP> .\Get-SPScripts.Copy-FilesToSP.ps1

    ReplyDelete
  78. Does not work, error always the same:
    "$web.Dispose()" throws "InvalidOperation"

    ReplyDelete
  79. Good Day Phil

    We wrote a console app that does something similar but unfortunately ours doesn't take sub folders, so really keen to use your script. Only problem is that we have a XML file per document as each document has unique attributes like Name and surname etc. Have you ever gotten round to allowing XML per file? Regards Phillip Conradie

    ReplyDelete
  80. This comment has been removed by the author.

    ReplyDelete
  81. This is just awesome!
    It would be even better if Jamie solution for CSV was published ;)

    ReplyDelete
  82. Hi Phillip,

    I need script to import CSV data as managed metadata into document library.


    ReplyDelete
  83. Is there a mechanism in your tool, one of your related tools, or another tool in Codeplex or elsewhere that will take the folder path, extract the path segments as metadata values, and apply those path segment metadata values as file metadata (either Enterprise Keyword or some governed term set)? How about taking the folder's and file's owner name and applying it to a metadata column as the file is copied to SP?

    ReplyDelete
  84. Is there a mechanism in your tool, one of your related tools, or another tool in Codeplex or elsewhere that will take the folder path, extract the path segments as metadata values, and apply those path segment metadata values as file metadata (either Enterprise Keyword or some governed term set)? How about taking the folder's and file's owner name and applying it to a metadata column as the file is copied to SP?

    ReplyDelete