Uploading Raw Data to Drupal 7

Using the Feeds Module for Bulk Import to Drupal 7

Here is how you can use the Feeds module to bulk import data from a CSV file to Drupal 7.  This covers file uploads and taxonomies too. This comes from Shakib Mostafa’s blog.

I once had a project where I needed to import a lot of museum collection data into Drupal from Access. This method worked pretty well.

The requirement:

Bulk data upload of a single content type into Drupal 7. The data will include text, dates, files and tags/taxonomies. There may be multiple tags/taxonomy terms per vocabulary.

The solution:

Use Feeds module to import data from CSV files as Drupal nodes.

Required modules:

  • Feeds
  • Feeds Tamper

Modules/submodules to enable:

  •  Feeds
  • Feeds Admin UI
  • Feeds Import
  • Feeds Tamper
  • Feeds Tamper Admin UI

Set up the CSV file:

Prepare the Excel file (or Access file) that you will convert to CSV – open up Excel and in the first row add the headers. Feeds importer requires a global ID to uniquely identify nodes. So the first column should contain this unique identifier. Name this column “guid“. The other column headers can be anything you like. For example, if you want to import a list of files with title, abstract, some tags and taxonomies; you can name your columns filename, title, description, tags, categories etc. The column headers do not have to match content type fields.

Add data to the Excel file:

  • guid: make sure each row has a unique number.
  • filename: You will have to upload all the files you want to migrate, into your Drupal root directory. So put all your files in a directory called import-uploads and upload them to your Drupal root. Make sure that the filenames have the path from your Drupal root. So each filename should have “import-uploads/” prepended. E.g. “import-uploads/file1.pdf”, “import-uploads/file2.avi”, etc.
  • title/description: should have text, make sure that your title is less than 256 characters.
  • tags: add all tags (actual words for tags) on the same column for a specific row. Make sure that tags are semicolon separated. For example, if you want to tag the same node with “funny” and “entertaining” the entry should be “funny;entertaining”.
  • taxonomies: This will probably require the most amount of effort. You have to find the taxonomy term ids for the terms you want to use for each row. Just like tags, if you have multiple taxonomy terms per node, make sure they are separated by semicolons. For example, if you have a taxonomy set up with terms “physics” with term id 22 and “math” with term id 25 and you want to classify the same node with both, the entry in the taxonomy column should be “22;25”.
  • Save as CSV: save the Excel file in CSV format – make sure to quote all cells. Otherwise you will run into errors if any of the text (e.g. description) contains commas.

Set up Feeds Importer:

  • Navigate to Structure -> Feeds Importer -> Add Importer.
  • Give your importer a meaningful name like “bulk file upload” and click Submit.

On the next screen follow these directions:

  • Basic Settings: leave as is, no changes needed.
  • Fetched: click “change” and select the “File Upload” option and click save.
  • Fetched Fileupload: click settings, make sure to add all the file extentions you want to add, e.g. pdf, wav, doc, docx etc.
  • Parser: click “change“, select the “CSV Parser” option and save.
  • CSV Parser: leave as is.
  • Processor: leave as is (Node Processor).
  • Node Processor: click settings, select “Replace existing node” for “Update exisiting node“, select the Content Type you want to use for this import. You may choose to use an existing user as the author.
  • Click Save.
  • Node Processor Mapping: This is where you will map your CSV file headers to the Content Type fields. One by one, add the column headers of the CSV file to the mapping and choose the appropriate Content Type field to map to.
  • Click Save.
  • Now make some configuration changes for multi value inputs such as tags or taxonomies. To do this click on the “configure feeds tamper” link near the top of the Add Importer page.
    For tags and taxonomies only, click “Add plugin“. Select “Explode” as the “plugin to add“, and in the “Configure Explodes” section change String Separator to a semicolon “;“. This is the configuration that tells the feeds importer that tag/taxonomies have multiple values in the CSV file that are separated by semicolons. Once you are done, click the “Add” button. And then save the Feeds Tamper configurations.

Run the import:

  • Navigate to Structure -> Feeds Importer, and click on the “import” link.
  • Click on the Bulk File Upload link.
  • Upload your CSV file into your sites/default/files folder.
  • Upload your import-upload folder, containing all the files to be imported into your Drupal root.
  • Enter your CSV file’s name in the File text field, with a the word “public://” prepended. So the entry should be something like “public://bulk_import.csv”.
  • Click on the Import Button.

If you run into isses, you will see error messages at the top. Often times, these issues arise from typos in the CSV file, e.g. filename on CSV does not match filename on file system. If you come across issues like these, make the changes in the CSV and rerun the Import. You can keep on rerunning the import on the same CSV file as many times as you want.