-
Tideways and Xhgui using DevDesktop and Docker
THIS POST IS UNFINISHED. Use at your own risk. I needed to share with a colleague, so I’m just getting it out into the world. Your Mileage May Vary!
I’ve been working on some large Drupal 8 migrations and have been wanting to profile them for some time due to a few migrations taking far more time than I expected. Acquia DevDesktop, which I happen to be using for this site, offers xhprof in the older PHP environments, but I wanted to get something setup in PHP 7.
For PHP 7 the recommendation is to use Tideways; it’s a modern fork of xhprof. After collecting (tracing/profiling) the data with Tideways you need a way to analyze the data. I used a Docker environment to get Xhgui running. Here’s what a few xhgui screens look like. The best part is that nearly everything is clickable so you can drill down to figure out what’s slow, and why!
-
Block Caching “Gotcha” in Drupal 8
I recently had a fight with the Block system in Drupal 8. To be brief, if you’re trying to disable caching on a block, make sure to set the #cache element regardless of whether the block has output or not.
This does not work (empty result gets cached):
1234567891011121314151617181920212223242526/*** {@inheritdoc}*/public function build() {$output = [];$user_id = \Drupal::routeMatch()->getRawParameter('user');$current_user_id = \Drupal::currentUser()->id();if ($user_id == $current_user_id) {$output = ['#theme' => 'item_list','#list_type' => 'ul','#items' => [Link::fromTextAndUrl(t('Edit profile'), Url::fromRoute('entity.user.edit_form', ['user' => $user_id])),Link::fromTextAndUrl(t('Logout'), Url::fromUserInput('/user/logout')),],'#wrapper-attributes' => ['class' => 'member-utilities'],'#cache' => ['max-age' => 0,],];}return $output;}This does work (nothing gets cached, as desired):
123456789101112131415161718192021222324/*** {@inheritdoc}*/public function build() {$output = ['#cache' => ['max-age' => 0,],];$user_id = \Drupal::routeMatch()->getRawParameter('user');$current_user_id = \Drupal::currentUser()->id();if ($user_id == $current_user_id) {$output['#theme'] = 'item_list';$output['#list_type'] = 'ul';$output['#items'] = [Link::fromTextAndUrl(t('Edit profile'), Url::fromRoute('entity.user.edit_form', ['user' => $user_id])),Link::fromTextAndUrl(t('Logout'), Url::fromUserInput('/user/logout')),];}return $output;} -
Send email after Migrate Import in Drupal 8
Here’s an example of a barebones event subscriber to send an email when a Drupal migration finishes importing. The success vs failure logic isn’t working correctly, but you’ll get the gist. If you come up with a solution for this please contact me; I don’t have time to work on it at the moment.
The file structure looks like this:
mymodule.services.yml
12345services:mymodule_migrate.subscriber:class: Drupal\mymodule_migrate\EventSubscriber\MymoduleMigrateSubscribertags:- { name: event_subscriber } -
Python Faker with the Fake App (Mac OS)
Fake App is a fantastic way to automate form filling, especially for folks without coding skills. The click-and-drag to target an element lets you quickly build your macros.
Python Faker is a utility library that I leverage with tools like Fake App, Alfred, Keyboard Maestro, etc. to generate realistic dummy text. This blog post shows one method of how to get Python Faker data into Fake App; I imagine there might be a simpler way but this works well and is fairly intuitive.
The idea is to just write the Faker output to a file and read that file into a variable for use in Faker. With this method you could even store variables for re-use. For example, if you had three fields: First Name, Last Name, and Full Name, you could easily store the first and last names to separate files and use both values for the Full Name field. With the power of Faker your dummy text options are extensive; you can generate very realistic values for your form filling. See all of the “providers” here.
This example below took just minutes to setup (and it was the first time I’ve used this technique).
-
Setting up xdebug for PHP 7 in Acquia DevDesktop
The Acquia DevDesktop help page says:
The PHP 7 version currently included with Acquia Dev Desktop does not currently include Xdebug. You can download an updated version of Xdebug here .
Here are the actual steps I used. YMMV.
123456789101112131415161718192021cd /Applications/DevDesktop/php7_0/extwget http://xdebug.org/files/xdebug-2.5.5.tgztar xvf xdebug-2.5.5.tgzrm xdebug-2.5.5.tgzcd xdebug-2.5.5/Applications/DevDesktop/php7_0/bin/phpize./configure --enable-xdebug CC="gcc -arch i386" CXX="g++ -arch i386" -with-php-config=/Applications/DevDesktop/php7_0/bin/php-configmakemv modules/xdebug.so ../cd ..rm -rf package.xml xdebug-2.5.5# Edit /Applications/DevDesktop/php7_0/bin/php.ini# Add these lines:zend_extension="/Applications/DevDesktop/php7_0/ext/xdebug.so"xdebug.remote_enable=1xdebug.remote_log=/tmp/xdebug.log# Restart DevDesktop services -
Drupal 8 Migrate Process Plugin: migration_lookup_target_ids
I was having some difficulties using the migration_lookup process plugin. I wrote this plugin so that I could make sense of how the migration lookup was working and the data that it was returning. The plugin ended up being something that I use after every migration_lookup as it seems to ensure error-free lookups. YMMV.
-
Drupal 8 Migrate – Tips and Tricks
This is an ongoing brain dump of quick tips for working with Drupal 8 Migrate. These items may seem true to me as I’m writing them today, but I’m constantly being reminded that things aren’t always as they seem with Migrate. It’s a wonderful system but can be tricky to understand. I recommend reading through my other Migrate-related posts, especially the post on debugging.
-
Extending the Migrate Plus JSON Parser in Drupal 8
The migrate_plus module provides a great JSON parser for easy migration of JSON data. With a small amount of code in a new class you can tweak the parser a bit. In my case I wanted to make a few simple changes:
- Speed up initial processing of the data
- Almost all of my data is at the top level in the JSON structure (there is no “Members” wrapper, for example).
- The stock JSON parser runs a selectByDepth() method if you use a item_selector: 0 , which slows things down considerably; it loops through every record, which is a bear if you have 46,000 records.
- Make it easy to manipulate the data (think prepare row)
- Using my overridden JSON parser as a base for additional parser classes it became nice to implement a simple prepareRows method to massage the data for each particular migration.
- I have several migrations coming out of the same data source (content type + supporting paragraphs). It’s nice to create a parser that extracts only the paragraph data so that we maintain an accurate count of the records for the “child” migration.
- Speed up initial processing of the data
-
Monitoring a Drupal 8 Migration Executed with Drush
Update 2 is proving to be the most useful method.
Update 1
Somehow I missed the documentation regarding the --feedback argument for the drush mi command. It’s similar to the methods below in that it helps you see that the migration is working, though you don’t see the totals. You can ask for feedback in seconds or items.
Update 2:
Another method that is a bit dirty but very useful is to add a single line to core/modules/migrate/src/MigrateExecutable.php.
Adding the highlighted line to \Drupal\migrate\MigrateExecutable::import will let you see exactly where the migration is (which row is being processed).
If this is something you want permanently, consider posting an issue on d.o to request more verbose logging. This is my hackey solution for my immediate needs.
123456789$destination = $this->migration->getDestinationPlugin();while ($source->valid()) {$row = $source->current();$this->sourceIdValues = $row->getSourceIdValues();error_log(date('c') . ' Processing row ' . $this->counter . ' of ' . $this->migration->id() . ': ' . json_encode($this->sourceIdValues));try {$this->processRow($row);$save = TRUE;The output (in your logs, and in the drush output) will be something like this (institution_node is my migration, inst_guid is the only id):
As of Thursday, August 16, 2018 the counter() method is removed. So, you have to drop a counter variable in place like this:
This method of monitoring execution is surprisingly effective as it’s fast and you don’t have to lift a finger for it to happen each time you run a migration.
Original Post
This isn’t exactly an ideal solution, but it will do in a pinch. I’m working on a migration of more than 46,000 nodes (in a single migration among many). I needed a way to monitor the progress but I wanted to do it as quickly as possible (no coding). Here’s what I came up with, which I run in a new iTerm window.
All we’re doing is asking how many records are in the “member_node” migration map every 5 seconds. If we started at 6350 items we know that by the end of 20 seconds we have created 35 records.
We could also query the target content type itself:
Here we can analyze the data the same way to see how many records the migration is creating.
I recognize that this is pretty crude, but it’s effective; you are able to see that it’s working, instead of just staring at an unchanging prompt and watching your CPU jump.
-
Alter Date Format in a Drupal 8 Migration
Per https://www.drupal.org/node/2820490 you can now use the format_date plugin to alter date formats like this:
1234567891011process:field_plist_date:-# Prevent entire row from being ignored if value is emptyplugin: skip_on_emptymethod: processsource: date-plugin: format_datefrom_format: 'n/j/y'to_format: 'Y-m-d'I do always recommend using skip_on_empty before any other process plugins.
