SharePoint 2010 Bulk Document Importer

New Post: ERROR - The document library does not exist

JPPowers — Thu, 16 May 2013 20:54:55 GMT

Did anyone figgure out if you can import files into sub folders of a document library ?

Closed Issue: Add support for claims authentication w/ windows [6963]

melg — Thu, 16 May 2013 10:40:12 GMT

It would be great to have support for claims authentication using the NT Token provider. Attempting an import with this configuration currently results in a 403 as the client attempts to redirect to the windows authentication based login page.

This blog post explains what's involved:
http://blogs.technet.com/b/speschka/archive/2010/06/04/using-the-client-object-model-with-a-claims-based-auth-site-in-sharepoint-2010.aspx

Thanks

Closed Issue: import fails after renaming a document library [7068]

melg — Thu, 16 May 2013 10:40:12 GMT

After renaming a document library the program incorrectly tries to save files to an url containing the new name.

Closed Issue: -CreateEmptyFolders flag condition is not implemented. [7130]

melg — Thu, 16 May 2013 10:40:11 GMT

Tried using the exe and searching the source, this flag/option does not seem to be implemented.

FYI.

New Post: timeout

FastFngrz — Tue, 23 Apr 2013 01:37:33 GMT

I see the same issue as well, Windows 7 x64 uploading to SP2013 Server - timeout on several files, always at 180 seconds.

New Comment on "Command-line options"

GoCart — Wed, 27 Mar 2013 18:20:58 GMT

The version I downloaded a couple of weeks ago lists this command-line argument: -MovedFolder - If present, all imported files will be moved to the specified folder Would you document this argument here as well, please? I don't seem to be giving it what it wants, or perhaps it doesn't really exist in the code itself...? Thanks!

Created Issue: How to add metadata to sharepoint site [9956]

Wafaa — Wed, 13 Mar 2013 09:23:51 GMT

Hi admin ,,,
I have toke this source code and tested it ,, and every things gone ok ,, but I want to import a metadata for example for a picture to my sharepoint 2010 site , how can I do it ??

B.R
Wafaa

New Post: CreatedBy and ModifiedBy not imported.

melg — Thu, 07 Mar 2013 14:51:05 GMT

Hi Tyler,

Great to hear that you used the tool with success, and thanks for sharing your improvements and thoughts. I am not working on this project anymore, but i'll contact some people with the suggestion to incorporate them in a new release.

New Post: Does it work with SharePoint Online?

finisterre — Thu, 07 Mar 2013 11:44:37 GMT

I take it this doesn't work with SharePoint Online? Doesn't seem possible to pass login credentials properly.

Commented Issue: Problem with illegal chars and blank [9932]

sluppo — Tue, 05 Mar 2013 15:03:18 GMT

I need to make a massive upload to SP from a shared disk, but uploader skips several files having filename starting with illegal chars followed by space.

eg: "~ A Cover Letter - Charter of Flotel.pdf"

"ERROR System.Net.WebException: The remote server returned an error: (414) REQUEST URI TOO LONG."

Your fantastic tool removes illegal but not the space.

Tnx
Sergio
Comments: ** Comment from web user: sluppo **

Solved by myself:

In Class FileNameConverter it is sufficient to invert order:
first replace illegal characters
next trim "." and " " from filename

bye
Sergio

Created Issue: Problem with illegal chars and blank [9932]

sluppo — Mon, 04 Mar 2013 16:28:24 GMT

New Post: timeout

bigben73 — Tue, 26 Feb 2013 12:47:38 GMT

Hello,

if I'm uploading a large file I always get a timeout on 180 seconds.
Can i increase this timeout.
I checked it with my Userinterface on my sharepoint account and it works.
Only with Sharepoint bulk importers i get this failure.

BR

New Post: Creation folder error

gregorywmitt2 — Thu, 07 Feb 2013 19:37:02 GMT

I made a test renaming the folder _vti_cnf and it works.

_vti_cnf is na invalid expression. or prohibited word?

New Post: CreatedBy and ModifiedBy not imported.

tschacht — Thu, 07 Feb 2013 18:59:21 GMT

Here I will describe some things I learned in my migration job using this application. I migrated over 300K files using this tool, so it definitely can hold up and do the job.

One key is to divide and conquer. My entire migration consisted of about 20 'sub-jobs'. It is impossible to get through a massive migration in one shot. You will run into roadblocks.

On very large jobs, it is important that you be able to restart where you left off should the program terminate in the middle of the job. To do this, I added a 'moveFileAsCompleted(file)' method into the SharePointRepository CreateFile() method right after the 'fileCreated = true' line. This meant that if my job error'd, I could identify and fix the error and immediatly start back where I left off. The moveFileAsCompleted simply moved the file that just uploaded into a 'completed' area outside of the path that the orbitone importer was working on. It looked something like:

        private void moveFileAsCompleted(ImportFile file) {
            var newPathFull = file.SourceFile.FullName.Replace("\\Version", "\\VersionMoved");
            var newPath = file.SourceFile.DirectoryName.Replace("\\Version", "\\VersionMoved");
            if (!Directory.Exists(newPath)) {
                Directory.CreateDirectory(newPath);
            }
            System.IO.File.Move(file.OriginalFullName, newPathFull);
            var metaFile = file.SourceFile.FullName + ".meta";
            if (System.IO.File.Exists(metaFile)) {
                System.IO.File.Move(metaFile, newPathFull + ".meta");
            }
        }

There is a filesize cap of 200MB by default in the code. If you need to override this, check out SharePointRepository.CreateValidator(). Here, you can also edit blocked file extensions and other options you may wish to alter.

If you wish to load multiple versions of a file, my approach was to have a filestructure representing each version. For example, if I had 200 files and 3 of those went all the way up to version 5, I would have a folder structure like so:

../Version1/(my base)
../Version2/(my base)
../Version3/(my base)
../Version4/(my base)
../Version5/(my base)

The Version1 folder would have all 200 of my documents. The Version2 folder would have any of those 200 which had at least two versions. The Version5 folder would have only the 3 files which actually had a version 5. With that folder structure in place, I would run a .bat file which would sequentially call the importer similar to the one below:


...\bin\Debug\OrbitOne.SharePoint.Importer.CommandLine.exe -site:"http://mysite.com/programmgmt"   -documentlibrary:"Program Documents"  -username:xxx  -password:xxx -domain:corp -CreateFolders  -folder:"E:\xml_export\programmgmt\Version1\"
...\bin\Debug\OrbitOne.SharePoint.Importer.CommandLine.exe -site:"http://mysite.com/programmgmt"   -documentlibrary:"Program Documents"  -username:xxx  -password:xxx -domain:corp -CreateFolders  -folder:"E:\xml_export\programmgmt\Version2\"
...\bin\Debug\OrbitOne.SharePoint.Importer.CommandLine.exe -site:"http://mysite.com/programmgmt"   -documentlibrary:"Program Documents"  -username:xxx  -password:xxx -domain:corp -CreateFolders  -folder:"E:\xml_export\programmgmt\Version3\"
...\bin\Debug\OrbitOne.SharePoint.Importer.CommandLine.exe -site:"http://mysite.com/programmgmt"   -documentlibrary:"Program Documents"  -username:xxx  -password:xxx -domain:corp -CreateFolders  -folder:"E:\xml_export\programmgmt\Version4\"
...\bin\Debug\OrbitOne.SharePoint.Importer.CommandLine.exe -site:"http://mysite.com/programmgmt"   -documentlibrary:"Program Documents"  -username:xxx  -password:xxx -domain:corp -CreateFolders  -folder:"E:\xml_export\programmgmt\Version5\"

As long as Version tracking is turned on in your doc library, each version should subsequently upload OK. I do recall that I had to alter the DocumentLibraryRepository.CreateFile() function to enable the upload over an existing file. Here is a version of that function which also includes some detailed error handling as well as some retry logic:


void CreateFile(ImportFile importFile, ClientContext context)
        {
            string serverRelativeFileUrl = string.Concat(m_serverRelativeListUrl, importFile.ServerRelativePath);

            using (var stream = importFile.OpenRead())
            {
                if (m_settings.Mode == ImportMode.Execute)
                {
                    log.Info("In CreateFile: " + ApplicationUrl + serverRelativeFileUrl);
                    int tries = 0;
                    bool successful = false;
                    int limit = 10;
                    while(tries < limit && !successful) {
                        try {
                            tries++;
                            // added true to overwrite since checks are done earlier anyways.....TS
                            log.Info("About to call File.SaveBinaryDirect: " + serverRelativeFileUrl);
                            File.SaveBinaryDirect(context, serverRelativeFileUrl, stream, true);
                            successful = true;
                        }
                        catch (Exception ex) {
                            if (ex.Message.ToLower().Contains("internal server error")) {
                                log.Info("Caught retryable File.SaveBinaryDirect exception for: " + serverRelativeFileUrl);
                            } else {
                                throw (ex); // force quit
                            }
                        }
                        if (tries >= limit) {
                            throw new Exception("Retried " + limit.ToString() + " times and still didnt succeed for: " +
                                          serverRelativeFileUrl);
                        }
                    }

                    log.Info("Succeeded ( " + tries + " attempt(s) ): " + serverRelativeFileUrl);
                }
            }
        }

Job validation: Part of my process was validating that each file was uploaded on SharePoint after the job had completed. My method to do this was to leverage my custom metadata. In my metadata, I had a field 'SharenetID' which contained a unique ID for each document. So, when my job was finished, I created a process which would read all my source files, parse their SharenetIDs into a giant list, and verify each of those IDs was found on a document in SharePoint. When this was done, I would be given a list of all docs which did not migrate successfully. I would then handle these on a case by case basis. Sometimes the problem would expose a bug in my code which I would fix. Sometime there would be an invalid filetype that failed upload. Sometime, I would just upload the file via the browser and be done with it. If the documents you are migrating do not have a unique identifier, you could create a script/app which simply appends an additional metadata value to the object with a unique GUID. Then, after importing, you will have a means to verify completion on a file-by-file basis.

The 'workhorse' block of code in my 'verifier' function is below. This is the code which will connect to a sharepoint library and pulldown all values of a given column (in my code below, it is SharenetID):

           //Dictionary<String, String> xmlIds =   .....my externally mapped list of IDs I am checking on SharePoint....

           List<string> spDocs = new List<string>(10000);

            using (var context = CreateContext(pArga)) {
                context.RequestTimeout = 10*60*1000;  // ten minutes
                Web web = context.Web;
                List list = context.Web.Lists.GetByTitle(pArga.SharePointLibrary);
                context.Load(context.Site);
                context.ExecuteQuery();
                context.Load(list);

                CamlQuery cq = new CamlQuery();

                cq.ViewXml =
                    "<View Scope='Recursive'><ViewFields><FieldRef Name='SharenetID'/></ViewFields></View>";

                ListItemCollection collListItem = list.GetItems(cq);

                context.Load(collListItem);

                context.ExecuteQuery();

                foreach (ListItem oListItem in collListItem) {
                    if (oListItem["SharenetID"] != null)
                        spDocs.Add(oListItem["SharenetID"].ToString());
                }


                foreach (var xid in xmlIds.Keys) {
                    if (!spDocs.Contains(xid)) {
                        log("=========================================");
                        log("Didn't find On SP: " + xid);
                        log(xmlIds[xid]);
                        var sizeString = "File not found....";
                        if (File.Exists(xmlIds[xid])) {
                            sizeString = xmlIds[xid].Length.ToString();
                        }
                        log(sizeString);
                    }
                }

            }

Performance: Performance was acceptable by default. I was able to increase perfomance by implementing multi-threading. I found best performance with around 5-8 threads. You could also simulate multithreading by running multiple instances of the exporter on different physical folders.

That's all I could think of at the moment. If there are any questions, please post in here and I will reply the best I can.

New Post: CreatedBy and ModifiedBy not imported.

tschacht — Thu, 07 Feb 2013 18:08:24 GMT

Hi all,

My company restrictions ended up preventing me from uploading me entire modified solution. However, I have recreated some demonstration classes to explain the bulk of the process I used to implement a AD user import as well as a general metadata import.

To be able to import domain users, each user must be an active user already on the sharepoint site. Your customemetadata provider (in the below example, SharenetCustomMetadataProvider), will use an instance of a IUserMapper (in the below example, SharenetUserMapper) to resolve username strings to their SharePoint equivelent user. The SharenetUserMapper below uses another class provided, CorpDomainUserRepository, to connect to the passed in SharePoint site and get a list of all SharePoint users on that site.

public class SharenetCustomMetadataProvider : IMetaDataProvider {

        private IUserMapper myUserMapper;

        public SharenetCustomMetadataProvider() {
            myUserMapper = new SharenetUserMapper(SettingsSingleton.GlobalSettings.SiteUrl);
        }

        public User GetAuthor(string filename) {
            var d = GetAllMetaData(filename);
            if (d.ContainsKey("CreatedBy"))
            {
                if (myUserMapper.Map(d["CreatedBy"].ToUpperInvariant()) != null) {
                    return myUserMapper.Map(d["CreatedBy"].ToUpperInvariant());
                }
            }
            return null;
        }

        public User GetEditor(string filename) {
            var d = GetAllMetaData(filename);
            if (d.ContainsKey("ModifiedBy")) {
                if (myUserMapper.Map(d["ModifiedBy"].ToUpperInvariant()) != null) {
                    return myUserMapper.Map(d["ModifiedBy"].ToUpperInvariant());
                }
            }
            return null;
        }

        private IDictionary<string, string> GetAllMetaData(string filename) {
            var d = ParseMetadata(filename);

            return d;
        }

        public IDictionary<string, string> GetMetaData(string filename) {
            var d = ParseMetadata(filename);
            if (d.ContainsKey("CreatedBy")) d.Remove("CreatedBy");
            if (d.ContainsKey("ModifiedBy")) d.Remove("ModifiedBy");
            return d;
        }

        private Dictionary<string, string> ParseMetadata(string diskPath) {

            var dict = new Dictionary<string, string>();
            var metadataFilename = "";

            if (Directory.Exists(diskPath)) {
                DirectoryInfo di = new DirectoryInfo(diskPath);
                metadataFilename = di.FullName + Path.DirectorySeparatorChar + "folder.meta";
            } else if (File.Exists(diskPath)) {
                FileInfo fi = new FileInfo(diskPath);
                metadataFilename = fi.FullName + ".meta";
            }


            if (!String.IsNullOrEmpty(metadataFilename) && File.Exists(metadataFilename)) {
                var lines = File.ReadAllLines(metadataFilename);
                foreach (string line in lines) {
                    if (line.Contains("=")) {
                        dict[line.Substring(0, line.IndexOf('='))] = line.Substring(line.IndexOf('=') + 1);
                    }
                }
            }


            //dict.ToList().ForEach(x => Console.WriteLine(x.Key + ":" + x.Value));
            return dict;

        }

    }

    public class SharenetUserMapper : IUserMapper
    {
        private string SiteCollectionUrl { get; set; }
        private CorpDomainUserRepository m_repository;
        private IDictionary<string, User> m_mappings;

        public SharenetUserMapper(string siteCollectionUrl) {
            SiteCollectionUrl = siteCollectionUrl;
        }

        public User Map(string username) {

            // lazy init
            if (m_repository == null) {
                m_repository = new CorpDomainUserRepository(SiteCollectionUrl);
                m_mappings = m_repository.GetUsers().ToDictionary(
                    user => user.Name.ToUpperInvariant(),
                    user => user
                    );
            }


            if (!m_mappings.ContainsKey(username.ToUpperInvariant())) {
                Console.WriteLine("can't find user: " + username.ToUpperInvariant());
                return null;
            }
            return m_mappings[username.ToUpperInvariant()];
        }
    }


    public class CorpDomainUserRepository
    {
        private string m_url;

        public CorpDomainUserRepository(string url) {
            m_url = url;
        }

        public IList<User> GetUsers() {
            using (var context = new ClientContext(m_url)) {
                var userList = context.Web.SiteUserInfoList;
                var users = userList.GetItems(new CamlQuery { ViewXml = "<View/>" });
                context.Load(users);
                context.ExecuteQuery();


                foreach (var user in users) {
                    //Console.WriteLine(user["Name"]);
                }

                var ulist = new List<User>();

                foreach (var user in users) {
                    ulist.Add(
                        new User {
                            Id = Convert.ToInt32(user["ID"]),
                            Name = user["Name"].ToString()
                        }

                        );
                }

                return ulist;
                //                return null;

            }
        }
    }

My particular implementation was to do like danieleg mentioned above: For each document imported into sharepoint, I had a corresponding metadata file in the same folder. For example, if I had Spreadsheet.xls, I would have an additional Spreadsheet.xls.metadata file with each line having a key/value pair seperated by an equals sign. One thing to remember is that you need to set these .metadata files as 'hidden', or the importer will import them into sharepoint as well. Here are the example contents of one of those files:

Description0=Emergency Support for Process
URL=https://sharenet.google.com/Open/389034796
BaseName=Emergency Support for Process
ModifiedBy=addomain\sptadmin
CreatedBy=addomain\johndoe
SharenetID=389034814

In the example above: URL and SharenetID are custom defined fields on my doc library. BaseName and Description0 correspond to the filename and description fields. ModifiedBy and CreatedBy correspond to AD users. The OrbitOne code tries to automatically map any fieldnames to a corresponding existing field on the sharepoint object.

If you look in the SharenetCustomMetadataProvider, the 'ParseMetadata()' method examines the passed in sharepoint doc file and loads in the corresponding metadata file should it exist. I also added support for attaching metadata to folders via a 'folder.meta' file that lives inside each folder.

FYI, the code that invokes the metadata file load is inside the FileSystemSource.Load() method.

To directly specify which custom metadata file to use, you can edit the constructor of FileSystemSource(). You can see in there where the FileSystemSource.MetaDataProvider object is being initialized.

If there are any questions, please leave them here. I will monitor the thread and answer the best I can.

New Post: Creation folder error

gregorywmitt2 — Thu, 07 Feb 2013 17:28:51 GMT

I have the following error when i execute a bulk import from a file server.

The user who execute bulk import have permission to create folders and upload files in the SharePoint library. This user doesn't have any quota configuration.

What is wrong?

This error is caused by variable overflow or something like this?

PS C:\spImporter> .\OrbitOne.SharePoint.Importer.exe -folder:"d:\teste2" -site:"http://docs.senacrs.com.br/sites/asti/" -documentlibrary:"teste" -createfolders
INFO - Configuring application
INFO - Initializing Sharepoint Connection
INFO - Initializing Validation
INFO - Initializing data source: File system: d:\teste2
INFO - Number of files: 2
INFO - Number of folders: 11
INFO - Number of empty folders: 10
INFO - Total file size: 0 bytes
INFO - Start Import

Unhandled Exception: Microsoft.SharePoint.Client.ServerException: Não é possível criar a pasta "sites/asti/teste/Desenvo
lvimento/Sistemas/ESTUDOS_TECNOLOGIAS E PROJETOS/Projetos/AdmNET/Codificação/Referencias/Site/Administracao/toolbar/_vti
_cnf".
at Microsoft.SharePoint.Client.ClientRequest.ProcessResponseStream(Stream responseStream)
at Microsoft.SharePoint.Client.ClientRequest.ProcessResponse()
at Microsoft.SharePoint.Client.ClientRequest.ExecuteQueryToServer(ChunkStringBuilder sb)
at Microsoft.SharePoint.Client.ClientRequest.ExecuteQuery()
at Microsoft.SharePoint.Client.ClientRuntimeContext.ExecuteQuery()
at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery()
at OrbitOne.SharePoint.Importer.SharePoint.DocumentLibraryRepository.CreateFolder(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.ImportContents(ImportFolder folder)
at OrbitOne.SharePoint.Importer.SharePoint.DefaultSharePointDestination.Import(ImportItem importItem)
at OrbitOne.SharePoint.Importer.DocumentImporter.Execute()
at OrbitOne.SharePoint.Importer.CommandLine.Program.Main(String[] args)

New Post: CreatedBy and ModifiedBy not imported.

danieleg — Thu, 07 Feb 2013 14:28:03 GMT

Hi Tyler can you send me a copy of your solution to handle metadata? Thanks dguarneri71@gmail.com I have to upload a directory structure with some files and any file has its own metadata.
I want to put the files and metadata in the same folder by creating a metadata file for each file to be loaded with a custom extension, for example ".metadata". I use a custom provider. Do you think it possible?

New Post: Unsupported value line argument format.

avi081265 — Fri, 21 Dec 2012 06:23:33 GMT

I downloaded the setup and running the application using following command line

OrbitOne.SharePoint.Importer.exe -folder:"\\201.21.55.15\Share_AK" site:http://sp2010vm1:37686/sites/MyRecordCenter/ -documentlibrary:AKTestDocLib

I am getting following errror, please let me know If I am missing anything. I tested I have the full access of shared folder and as well as on site also.

2012-12-21 11:46:05,111 ERROR System.ArgumentException: Unsupported value line argument format.
Parameter name: site:http://sp2010vm1:37686/sites/MyRecordCenter
   at OrbitOne.SharePoint.Importer.CommandLineParsing.CommandLineDictionary.AddArgument(String argument)
   at OrbitOne.SharePoint.Importer.CommandLineParsing.CommandLineDictionary.FromArguments(IEnumerable`1 arguments, Char keyCharacter, Char valueCharacter)
   at OrbitOne.SharePoint.Importer.CommandLineParsing.CommandLineParser.ParseArguments(Object valueToPopulate, IEnumerable`1 args, Char keyCharacter, Char valueCharacter)
   at OrbitOne.SharePoint.Importer.CommandLine.Program.GetSettings(IEnumerable`1 args)
2012-12-21 11:46:05,131 ERROR documentlibrary is required
2012-12-21 11:46:05,131 ERROR site url is required
2012-12-21 11:46:05,132 ERROR folder is required
2012-12-21 11:46:05,142 ERROR Settings are not valid, Import aborted.

New Post: does it work for sharepoint 2007?

melg — Wed, 24 Oct 2012 09:01:34 GMT

No, SharePoint 2010 only. It uses the client object Model which was introduced with SharePoint 2010

New Post: does it work for sharepoint 2007?

tinkoo99 — Tue, 23 Oct 2012 17:33:48 GMT

does it work for sharepoint 2007?