How to Create Feeds

This is an advanced tutorial, that assumes at least some familiarity with Drupal content types, fields, and views.

Things Feeds are helpful for

  1. Creating news feeds or event feeds, so that content with the appropriate tag can be imported from Western Today onto a department site, or from a department site into the University Calendar
  2. Moving content from one Drupal site to another
  3. Keeping content synced between two different Drupal sites

Overview of the process

I'll refer to the site you're exporting content from as the export site, and the site you're importing to the import site.

In general, how feeds works is that you:

  1. Create a view on the export site in some kind of RESTful data format, in this case JSON or XML
  2. Use the feeds module on the import site to to map the fields from that view into nodes on the import site

This tutorial's example uses JSON as the RESTful format for most of the examples, but you can use XML if you prefer working with that data format.

To create a view, you'll need to both have login access to the export site and appropriate permissions. If you do not have access to the site (for example, if you are attempting to create a Western Today news feed) you'll need to request help from WebTech in getting the view set up. You can also request help with getting permissions configured if you do have access to the export site, but are unable to create the view.

Enable Modules

On both the import and export site, enable the module RESTful Web Services

On the import site, enable Feeds, Feeds Extensible Parsers, Feeds Tamper, and Tamper

On the Export Site: Set up a RESTful Export View

Create the RESTful export view

Create a view of the content type that you are exporting.

  • The view should be of the content type you are exporting.
  • Select Provide a REST export and set the REST export path

Tip: Number of Items to Display

By default, the view will be set up to display 10 items. It is a good idea to keep this limit while you are configuring and testing your feed. This keeps the import file small and loading quickly, and also gives you a small amount of data to troubleshoot with.

After you successfully have 10 items importing as expected, you can change or remove this limit to import more or all items.

Screenshot of "Add a new view" screen, with the settings set as described by the text preceding this image

Configure the View

Under the format section, open the Settings next to Serializer.

Screenshot showing "Settings" next to "Serializer" highlighted

Select the format you wish to use, JSON or XML.

screenshot showing JSON checkbox selected

Under Show, open Fields.

Next to Show: Fields is highlighted

Select Fields as the row style.

In the fields modal, the Fields radio is selected

Next to fields, open the Settings link. This will show you any available fields you have added in the next section of the view. For the Title, check the Raw output option. You may need to check this option for other fields in your view as you add them.

Next to fields, the Settings link is highlighted
In the field settings, the "raw output" checkbox is selected

Preview the View Output

Open the URL path that you set in your browser. Your browser should detect and parse the data as JSON or XML and display it in an easy to read and navigate format.

view setting showing the path set when the view was created, in this case /my-export-url

If you are unable to reach this URL or are served an error, double-check the settings from the previous section.

The path from the view typed out into the browser's URL bar, in this case https://mysite.wwu.edu/my-export-url
The JSON data as rendered by the browser, with each node as a separate line item

On the Import Site: Create a Feed

Recreate the content type

On the import site, create the the content type you'd like to import the data into. For example, if this is news feed, create a "News" content type. If you are migrating content, the content type will likely be the same as it was on the export site.

The fields you will need to add on the new content type should match the fields on your export view, and be the appropriate field type to hold that data.

Screenshot of the data output, which has fields that match those on the import site with a note pointing out that fields with number are being imported into an integer field type, and fields with links are being imported into a link field type

Create a feed type

Structure > Feed Type > Add Feed Type

  • Set the Fetcher to Download from url
  • Parser should be JsonPath (or XML)
  • Processor is usually Node (Though, you can import other entities, like users! See the special note at the end for User Feeds)
  • Set the content type to whatever you created

Under Settings:

  • Set the import period to whatever makes sense. For things like News and Events feeds, you'll want this to be somewhere in the 15min - 1hr range, as those updates should come through often. For things like content migrations, I usually set this to Never so that I'm only ever doing manual imports.

Under Fetcher Settings:

  • Check Always Download

Under Processor Settings:

  • Check Update existing contents
  • You may also want to check Force update under advance settings.

Then click the Save and add Mappings button.

Fetcher settings, with "Always Download" checked
The "add new feed type" screen, with configurations set as in the text

Set the Context & Mappings

Importing from a Drupal RESTful View

If you are using JSON, set the context to $.*

If you are using XML, set the context to //node

Importing from another source

If you are importing from an XML or JSON source that is not from a RESTful view as configured in these instructions, you may need to use another context. See the help link on that field for ideas on how to format this.

A screenshot of the first few mappings, showing that the title maps to title and is set as 'unique', with a special note that the 'context' should be different for xml views

Next you will create mappings. These mappings tell the feed to take a field from the JSON file, and associate it's data with a field in Drupal.

First, Select a Target. This is the Field you are mapping to.

A Source drop down will then appear. Select New xpath source

A text box will appear. In this textbox, put the field name as it appears in the JSON file.

Tip: Test early and Often

It's tempting to create all your mappings at once, but you will have a much easier time troubleshooting your feed and making sure the context is set correctly if you start with 1-2 fields first. After this, add a few fields at a time, importing after every couple of fields.

Essential: Unique Mapping

You will need at least one unique mapping. This unique mapping is how feeds keeps track of nodes. If content on the export site changes, feeds can update existing nodes on the import site with the unique mapping. If it's missing, feeds will create duplicated content.

A good practice for this is to use the NID or GUID from your export source, and import it into the Feeds Item GUID target.

Create the Feed and Test

To test your feed, you first need to... create it! All we've done so far is create the feed type, but not the feed itself.

So go to Content > Feeds > Add Feed. Name this feed and set the URL, which is the path to your JSON or XML document that you created in the view on your export site.

Click Save and Import.

screenshot of the create feed screen, showing the path matches that created in the view

If all goes according to plan, your feed should think for a moment, and create some new nodes. If you look at those nodes, they should have the data from the export site!

Success status message of "The feed has not been updated" and "There are no new online courses," with a table flipping emoji below that and the text "But there should be 10 new online courses!!!"

To be real, things almost never go according to plan at this point.

Thankfully, the errors that get thrown at this point are generally pretty good hints at where things went wrong. Sometimes it's something with the view, and sometimes it's an issue with mappings. You may also need to check the reports under Reports > Recent Log Messages for clues.

The best advice I can offer is keep tweaking things on your view and in your mappings, double check your context, and keep importing until things start working. There are a lot of details involved in the process, so a second set of eyes never hurts either.

Finishing up

There are only a few steps left at this point:

  • Continue adding mappings until you have all of your fields
  • Once everything seems to be working correctly, go back to your original view and change it from only displaying 10 items to displaying all items. Remember, no pagers!

When everything is configured, you can bring over hundreds or thousands of nodes in mere moments. Pat yourself on the back, you've done a great job.

Feeds Tamper

Feeds tamper is a useful tool for altering data at the point where it gets imported, but before the node gets created. Some really common uses for it:

  • Strip out white space before/after text
  • Decode HTML, for example when things like, "See & be seen" should be changed to "See & be seen"
  • Find and replace, say when you're changes a field that reads "Dr." to instead read "PhD"
  • Import plain text time/date formats

And all sorts of other cool things.