Extending the Migrate Plus JSON Parser in Drupal 8
The migrate_plus module provides a great JSON parser for easy migration of JSON data. With a small amount of code in a new class you can tweak the parser a bit. In my case I wanted to make a few simple changes:
- Speed up initial processing of the data
- Almost all of my data is at the top level in the JSON structure (there is no “Members” wrapper, for example).
- The stock JSON parser runs a selectByDepth() method if you use a item_selector: 0 , which slows things down considerably; it loops through every record, which is a bear if you have 46,000 records.
- Make it easy to manipulate the data (think prepare row)
- Using my overridden JSON parser as a base for additional parser classes it became nice to implement a simple prepareRows method to massage the data for each particular migration.
- I have several migrations coming out of the same data source (content type + supporting paragraphs). It’s nice to create a parser that extracts only the paragraph data so that we maintain an accurate count of the records for the “child” migration.
Here’s the current version of my base class. This will probably change over time, but I’ll try to keep this post updated. I have highlighted the rows that are different from the stock JSON parser.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
<?php namespace Drupal\mymodule_migrate\Plugin\migrate_plus\data_parser; use Drupal\migrate_plus\Plugin\migrate_plus\data_parser\Json; /** * Obtain JSON data for migration. * * @DataParser( * id = "json_mymodule", * title = @Translation("JSON Parser for My Module") * ) */ class JsonMymodule extends Json { /** * Retrieves the JSON data and returns it as an array. * * @param string $url * URL of a JSON feed. * * @return array * The selected data to be iterated. * * @throws \GuzzleHttp\Exception\RequestException */ protected function getSourceData($url) { $response = $this->getDataFetcherPlugin()->getResponseContent($url); // Convert objects to associative arrays. $source_data = json_decode($response, TRUE); // If json_decode() has returned NULL, it might be that the data isn't // valid utf8 - see http://php.net/manual/en/function.json-decode.php#86997. if (is_null($source_data)) { $utf8response = utf8_encode($response); $source_data = json_decode($utf8response); } if ($this->itemSelector === 0) { // Don't bother with selectByDepth (it's expensive, and we already // have the top-level data as desired). } elseif (is_int($this->itemSelector)) { // Backwards-compatibility for depth selection. $source_data = $this->selectByDepth($source_data); } else { // Otherwise, we're using xpath-like selectors. $selectors = explode('/', trim($this->itemSelector, '/')); foreach ($selectors as $selector) { if (!empty($selector)) { $source_data = $source_data[$selector]; } } } $modified_data = $this->prepareRows($source_data); return $modified_data; } /** * Modify any of the rows in the file. * * Any classes that implement JsonMymodule can simply declare * a protected prepareRows function and massage the data as needed * before returning it. * * @param array $source_data * Array of data. * * @return array * Modified data. */ protected function prepareRows(array $source_data) { // Do things with the $source_data! return $source_data; } } |
One Comment
Pisith Yim
Hi Adam,
How are you? My name is Pisith Yim, I am new to Drupal. So I am doing the exact same thing you’re doing trying to extend the migrate_plug module, I have no success on getting my custom functionality to fire instead of migrate_plus functionality. Any tips would be appreciated.
Best,
Pisith